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ABSTRACT 


This  thesis  focuses  on  analyzing  factors  that  affect  a  student’s  time  to  train  (TTT) 
as  he/she  completes  the  Joint  Strike  Fighters  (JSF)  pilot  training  syllabus  at  the  Integrated 
Training  Center  (ITC)  on  Eglin  Air  Force  Base.  It  identifies  the  most  robust  course  of 
action  (COA),  provides  accurate  TTT  estimates,  and  identifies  a  watch  list  of  factors  that 
have  the  most  effect  on  TTT.  The  results  of  this  thesis  provide  a  basis  and  a  justification 
for  procurement  and  resource  decisions  for  the  JSF.  They  also  provide  a  means  to  ensure 
a  proper  flow  of  pilots  to  operational  commands,  reduce  unnecessary  flights,  and  enable 
the  ITC  to  complete  its  mission  while  remaining  within  monthly  flight  time  and  airframe 
limits.  It  can  also  help  predict  the  impact  of  future  resource  decisions,  and  assist 
commanders  in  mitigating  their  operational  effect. 

The  primary  recommendation  for  reducing  TTT  is  to  have  students  work  1 1  hours 
per  day  and  have  the  workday  start  at  or  after  0900.  Increasing  the  number  of  full  motion 
simulator  (FMS),  while  reducing  TTT,  is  not  justified.  Although  a  limited  resource, 
airspace  is  not  a  potential  bottleneck  in  the  system  and  any  change,  short  of  elimination, 
has  very  little  influence  on  TTT.  Flight  refly  rates  and  aircraft  failure  rates  above  12% 
and  13%  respectively  significantly  increase  TTT  and  must  be  carefully  monitored. 
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EXECUTIVE  SUMMARY 


The  United  States’  current  inventory  of  fighter/attack  aircraft  is  up  against 
airframe  time  limits  in  the  not  too  distant  future.  The  United  States  Navy  estimates  a 
peak  shortfall  of  100  to  200  aircraft  by  2018  (O’Rourke,  2009).  A  new,  low-cost,  highly 
capable,  and  advanced  replacement  needs  to  be  produced. 

In  the  early  90s,  the  Joint  Advance  Strike  Technology  Program  (JAST)  was 
created  to  bring  together  and  mature  the  required  technologies  to  create  a  new  strike 
fighter  capable  of  meeting  the  requirements  of  the  Navy,  Marines,  and  Air  Force 
(Bolkcom,  2007).  In  1994,  Congress  mandated  that  the  JAST  program  be  merged  with 
the  Advance  Short  Takeoff  Vertical  Landing  program  and  the  Joint  Strike  Fighter  (JSF) 
was  bom. 

Eventually,  two  manufacturers  were  selected  to  compete  for  the  JSF  contract. 
Boeing  and  Lockheed  Martin  began  a  4-year-long  concept  and  demonstration  phase  that 
included  producing  a  flying  prototype.  The  competition  ended  in  October  2001,  when 
Lockheed  Martin  won  the  competition  with  its  X-35  prototype.  The  JSF  is  now  known  as 
the  F-35. 

The  F-35  is  produced  in  three  variants.  The  F-35A  is  used  by  the  Air  Force  and  is 
a  conventional  takeoff  and  landing  (CTOL)  aircraft.  It  is  a  replacement  for  the  F-16  and 
A-10,  and  will  complement  the  F-22  for  the  Air  Force’s  high-low  mix  (Gertler,  2010). 
The  F-35B  is  used  by  the  Marines  and  is  short-takeoff-and-vertical-landing  capable;  a 
requirement  for  use  on  amphibious  ships.  It  will  replace  the  AV-8  Harriers  and  older 
generation  F-18s  for  the  Marine  Corps  (Gertler,  2010).  The  F-35C  is  used  by  the  Navy;  it 
has  foldable  wings  and  is  aircraft-carrier-launch-and-recovery  capable.  The  three 
variants  have  a  70%  to  90%  commonality  in  parts  (Gertler,  2010).  This  is  an  inherent 
design  philosophy  to  reduce  manufacturing  cost  and  serviceability,  and  made  the  project 
truly  joint. 

Unlike  the  F-22  Raptor,  the  F-35  is  designed  not  just  to  be  joint,  but  also 
multinational.  To  date,  at  least  eight  other  countries,  ranging  from  Australia  to  Turkey, 


are  expected  to  be  partners  in  the  production  and  procurement  process  (Bolkcom,  2007). 
One  thing  all  the  nations  will  have  in  common  is  that  their  pilots  and  maintainers  will  all 
initially  be  trained  in  the  United  States.  The  Integrated  Training  Center  (ITC)  is  the 
multiservice,  multinational  schoolhouse  for  training  personnel  to  pilot  and  maintain  the 
F-35.  The  first  ITC  is  located  on  Eglin  Air  Force  Base  in  the  Florida  Panhandle.  More 
training  centers  will  be  built  as  additional  squadrons  come  on  line  and  more  aircraft  enter 
service.  The  ITC  has  all  the  resources  necessary  to  train  U.S.  and  allied  partners  in  flying 
and  maintaining  the  F-35.  It  consists  of  an  academic  and  operational  side.  The 
classrooms,  simulators,  and  computer-based  trainers  are  located  on  the  academic  side  of 
the  ITC,  while  the  aircraft,  brief  rooms,  and  maintenance  facilities  are  located  on  the 
squadron  side  of  the  ITC.  The  Navy,  Marine  Corps,  and  Air  Force  have  separate 
squadrons  and  instructor  pilots,  but  share  all  the  resources  that  make  up  the  ITC. 

In  order  to  procure  the  correct  number  of  resources  (simulators,  classrooms, 
instructors,  etc.)  for  the  ITC,  Lockheed  Martin  built  a  model  to  simulate  the  flow  of 
students  through  the  training  syllabus.  By  adjusting  parameter  levels,  along  with  a  set  of 
ground  rules  and  assumptions,  student  throughput  can  be  studied  to  see  their  effects  on 
metrics  such  as  time-to-train  (TTT)  and  resource  utilizations  rates.  These  results  are 
analyzed  and  used  to  justify  the  number  and  type  of  each  resource  needed  to  satisfy  the 
metrics  and  meet  any  requirements  as  specified  by  the  end  user.  The  goal  of  the  ITC  is  to 
provide  fully  trained  pilots  and  maintainers  to  meet  the  security  needs  of  the  future,  as 
stated  in  the  Joint  Contract  Specification  document  for  the  JSF:  “The  JSF  Air  System 
shall  provide  throughput  training  capacity  to  meet  both  peacetime  and  wartime  service 
requirements  for  pilots  and  maintainers”  (Born,  2007,  p.  8). 

The  ITC  model  simulates  day-to-day  ground  and  flight  operations  at  the  ITC.  The 
model  starts  collecting  data  when  students  and  resources,  such  as  aircraft  and  flight 
simulators,  are  inducted  into  the  system  and  ends  when  the  last  student  graduates.  Each 
student  has  a  syllabus  that  lists  all  classes,  meetings,  and  required  activities  that  need  to 
be  finished  before  that  student  can  graduate.  Upon  graduation  of  the  last  student,  the  ITC 
model  produces  detailed  reports  of  each  student’s  experience.  The  time  they  start  and 
finish  each  event  is  recorded,  as  well  and  any  delay  and  reason  for  delay  of  any  syllabus 
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event.  Resource  utilization  rates  are  recorded  and  used  as  justification  for  procurement  or 
as  an  explanation  for  bottlenecks  in  the  system. 

The  ITC  model  is  designed  to  run  in  either  “steady  state”  or  “ramp  up”  mode. 
The  ramp  up  mode  is  useful  in  predicting  the  impacts  of  different  resource  schedules. 
The  resources  that  can  be  varied  in  the  ramp-up  mode  are  the  number  of  full  motion 
simulators  (FMSs),  training  aircraft,  and  instructor  pilots.  These  inputs  are  provided  to 
the  model  in  the  form  of  a  text  file  containing  a  column  for  the  week  and  a  column  for  the 
change  in  number  of  that  resource.  This  is  the  mode  that  most  accurately  describes  the 
ITC  as  it  is  stood  up  and  is  the  one  that  is  studied  in  this  thesis.  Figure  1  summarizes  the 
process  up  to  the  analysis. 


Figure  1.  Integrated  Training  Center  Overview  Diagram  (From:  Born,  2007) 


The  objective  of  this  thesis  is  to  provide  the  Marine  Corps  with  a  consolidated 
plan  that  maximizes  their  resources  and  produces  pilots  for  the  JSF  as  quickly  as  the 
existing  constraints  will  allow.  With  JSF  production  delays  and  the  imminent  loss  of 
legacy  aircraft,  the  Marines  will  soon  be  facing  a  fighter  gap  (O’Rourke,  2009).  To  keep 
readiness  high,  the  Marine  Corps  must  be  able  to  depend  on  its  organic  air-to-air  and 
air-to-ground  capability  in  an  increasingly  complex  threat  environment.  This  all  hinges 
on  the  deployment  of  operational  squadrons  in  a  timely  manner,  and  the  efficient  and 
correct  usage  of  resources  at  the  ITC. 
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This  thesis  explains  and  demonstrates  a  method  to  develop  a  best  COA, 
identifying  the  resources  having  the  most  effect  on  reducing  TTT.  It  also  identifies  the 
resources,  and  the  limits  or  levels  of  these  resources,  which  have  the  most  effect  on  TTT. 
The  impact  of  aircraft  upgrades,  the  availability  of  training  airspace,  and  changes  to  the 
students’  induction  schedule  and  their  effects  on  TTT  are  analyzed. 

The  primary  motivation  for  this  research  is  to  help  the  Marine  Fighter  Attack 
Training  Squadron  (VMFAT-501)  to  effectively  and  efficiently  train  the  next  generation 
of  pilots  and  maintainers.  Communication  with  the  squadron  was  essential  and  ongoing 
throughout  the  research,  particularly  in  the  design  of  each  experiment.  Their  input 
narrowed  the  focus  of  the  research  and  helped  identify  the  most  important  aspects  of  the 
problem  for  further  investigation.  They  were  also  important  in  validating  the  results  from 
the  perspective  of  the  final  customer. 

The  overall  recommendation  for  reducing  TTT  is  to  have  students  work  1 1  hours 
per  day  and  have  the  workday  start  at  or  after  0900.  Increasing  the  number  of  FMS  to 
five  or  six,  while  reducing  TTT,  is  not  justified.  Likewise,  increasing  the  hours  the  ITC 
remains  open  has  no  effect  on  reducing  TTT  and  should  be  ignored.  The  current  plan  of 
having  four  aircraft  reserved  for  training  events,  while  the  excess  undergo  upgrades,  is 
the  best  and  most  robust  COA  to  ensure  a  minimum  TTT.  The  availability  of  airspace  for 
training,  short  of  eliminating  this  resource,  did  not  present  a  bottleneck  to  training  and  did 
not  change  TTT. 

The  results  produced  by  the  ITC  are  inherently  optimistic  due  to  known 
deficiencies  in  how  the  model  treats  weather,  maintenance,  other  stochastic  elements,  and 
in  the  strict  scheduling  of  student  events  (Dummar,  2011).  The  ITC  model  uses  means 
rather  than  distribution  for  its  random  variables,  this  also  gives  optimistic  results.  As  a 
consequence,  any  reductions  in  TTT  gained  for  choosing  the  COAs  recommended  in  this 
thesis  are  also  optimistic.  These  findings  do  provide  a  relationship  between  TTT  and  the 
controllable  and  uncontrollable  factors.  It  is  up  to  the  commander  and  training  officers  to 
monitor  these  factors  and  use  the  COAs  to  help  mitigate  their  effects. 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  United  States’  current  inventory  of  tighter/attack  aircraft  will  be  up  against 
airframe  time  limits  in  the  not  too  distant  future.  The  United  States  Navy  estimates  a 
peak  shortfall  of  100  to  200  aircraft  by  2018  (O’Rourke,  2009).  A  new,  low-cost,  highly 
capable,  and  advanced  replacement  needs  to  be  produced.  The  research  into  the 
replacement  of  the  current  inventory  of  aging  third  and  fourth  generation  aircraft  has 
been  ongoing  through  the  1980s  and  1990s,  with  no  airframe  making  it  to  the  production 
line.  The  early  stages  of  the  search  for  a  new  aircraft  had  the  Navy,  Marines,  and 
Air  Force  seeking  their  own  replacement,  with  different  aircraft  tailored  to  specific  needs. 
In  the  early  90s,  the  Joint  Advance  Strike  Technology  Program  (JAST)  was  created  to 
bring  together  and  mature  the  required  technologies  to  create  a  new  strike  fighter  capable 
of  meeting  the  requirements  of  the  Navy,  Marines,  and  Air  Force  (Bolkcom,  2007).  In 
1994,  Congress  mandated  that  the  JAST  program  be  merged  with  the  Advanced  Short 
Takeoff  Vertical  Landing  (ASTOVL)  program  and  the  JSF  was  born.  ASTOVL  was  a 
project  begun  in  1983  by  the  Defense  Advanced  Research  Agency  to  replace  the  aging 
AV-8  Harrier.  It  was  designed  to  be  a  multiservice  replacement  aircraft  and  fit  well  into 
the  framework  of  the  JAST  program. 

Eventually,  two  manufacturers  were  selected  to  compete  for  the  JSF  contract. 
Boeing  and  Lockheed  Martin  began  a  4-year-long  concept  and  demonstration  phase  that 
included  producing  a  flying  prototype.  The  competition  ended  in  October  2001,  when 
Lockheed  Martin  won  the  competition  with  its  X-35  prototype.  The  JSF  is  now  known  as 
the  F-35. 

The  F-35  will  be  produced  in  three  variants.  The  Air  Force  variant,  the  F-35A,  is 
a  conventional  takeoff  and  landing  (CTOL)  aircraft.  It  is  a  replacement  for  the  F-16  and 
A-10,  and  will  complement  the  F-22  for  the  Air  Force’s  high-low  mix  (Gertler,  2010). 
The  Marine  variant,  the  F-35B,  is  short-takeoff-and-vertical-landing  capable;  a 
requirement  for  use  on  amphibious  ships.  It  is  the  most  radical  design  of  the  three 
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variants  and,  thus  far,  the  most  prone  to  setback  (Majumdar,  2011).  It  will  replace  the 
AV-8  Harriers  and  older  generation  F-18s  for  the  Marine  Corps  (Gertler,  2010).  The 
Navy  variant  is  the  F-35C,  which  has  foldable  wings  and  is  aircraft-carrier-launch-and- 
recovery  capable.  It  is  physically  much  larger  than  the  two  previous  versions,  due  mainly 
to  the  requirement  for  airframe  reinforcements  because  of  the  additional  stress  of  catapult 
launches  and  arrested  landing.  It  also  has  a  larger  wing  area,  due  to  the  low  speed 
approach  profile  required  for  a  carrier-arrested  recovery.  The  three  variants  have  a  70% 
to  90%  commonality  in  parts  (Gertler,  2010).  This  is  an  inherent  design  philosophy  to 
reduce  manufacturing  costs  and  serviceability,  and  made  the  project  truly  joint. 

Unlike  the  F-22  Raptor,  the  F-35  is  designed  not  just  to  be  joint,  but  also 
multinational.  To  date,  at  least  eight  other  countries,  ranging  from  Australia  to  Turkey, 
are  expected  to  be  partners  in  the  production  and  procurement  process  (Bolkcom,  2007). 
One  thing  all  the  nations  will  have  in  common  is  that  their  pilots  and  maintainers  will  all 
initially  be  trained  in  the  United  States.  The  Integrated  Training  Center  (ITC)  is  the 
multiservice,  multinational  schoolhouse  for  training  personnel  to  pilot  and  maintain  the 
F-35.  The  first  ITC  is  located  on  Eglin  Air  Force  Base  in  the  Florida  Panhandle.  More 
training  centers  will  be  built  as  additional  squadrons  come  on  line  and  more  aircraft  enter 
service.  The  ITC  has  all  the  resources  necessary  to  train  U.S.  and  allied  partners  in  flying 
and  maintaining  the  F-35.  It  consists  of  an  academic  and  operational  side.  The 
classrooms,  simulators,  and  computer-based  trainers  are  located  on  the  academic  side  of 
the  ITC,  while  the  aircraft,  brief  rooms,  and  maintenance  facilities  are  located  on  the 
squadron  side  of  the  ITC.  There  will  be  three  squadrons,  one  for  each  variant  of  the  JSF. 
The  Navy,  Marine  Corps,  and  Air  Force  have  separate  squadrons  and  instructor  pilots, 
but  share  all  the  resources  that  make  up  the  ITC. 

In  order  to  procure  the  correct  number  of  resources  (simulators,  classrooms, 
instructors,  etc.)  for  the  ITC,  Lockheed  Martin  built  a  model  to  simulate  the  flow  of 
students  through  the  training  syllabus.  By  adjusting  parameter  levels,  along  with  a  set  of 
ground  rules  and  assumptions,  student  throughput  can  be  studied  to  see  their  effects  on 
metrics  such  as  time-to-train  (TTT)  and  resource  utilizations  rates.  These  results  are 
analyzed  and  used  to  justify  the  number  and  type  of  each  resource  needed  to  satisfy  the 
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metrics  and  meet  any  requirements  as  specified  by  the  end  user.  The  goal  of  the  ITC  is  to 
provide  fully  trained  pilots  and  maintainers  to  meet  the  security  needs  of  the  future,  as 
stated  in  the  Joint  Contract  Specification  document  for  the  JSF:  “The  JSF  Air  System 
shall  provide  throughput  training  capacity  to  meet  both  peacetime  and  wartime  service 
requirements  for  pilots  and  maintainers”  (Born,  2007,  p.  8). 

B.  PURPOSE 

The  objective  of  this  thesis  is  to  provide  the  Marine  Corps  with  a  consolidated 
method  that  maximizes  their  resources  and  produces  pilots  for  the  JSF  as  quickly  as  the 
existing  constraints  will  allow.  With  JSF  production  delays  and  the  imminent  loss  of 
legacy  aircraft,  the  Marines  will  soon  be  facing  a  fighter  gap  (O’Rourke,  2009).  To  keep 
readiness  high,  the  Marine  Corps  must  be  able  to  depend  on  its  organic  air-to-air  and  air- 
to-ground  capability  in  an  increasingly  complex  threat  environment.  This  all  hinges  on 
the  deployment  of  operational  squadrons  in  a  timely  manner,  and  the  efficient  and  correct 
usage  of  resources  at  the  ITC.  To  address  these  issues,  this  thesis  explains  and 
demonstrates  a  method  to: 

•  Develop  a  best  Course  of  Action  (COA),  identifying  the  resources  having 
the  most  effect  on  reducing  TTT. 

•  Identify  key  resources,  and  the  limits  or  levels  of  these  resources  that  have 
the  most  effect  on  TTT. 

•  Examine  airspace  issues  in  the  model.  Current  analysis  has  no  limits  on 
the  airspace  available.  Airspace  used  for  training  can,  and  often  needs  to 
be,  scheduled  in  advance.  Some  airspace  is  affected  by  prevailing  weather 
conditions  and  may  not  be  available  for  all  training  flights  equally.  A 
constraint  on  the  availability  of  this  resource  will  increase  TTT. 

•  Analyze  the  impact  of  aircraft  upgrades  on  the  overall  student  TTT. 

The  primary  motivation  for  this  research  is  to  help  Marine  Fighter  Attack 
Training  Squadron  501  (VMFAT-501)  to  effectively  and  efficiently  train  the  next 
generation  of  pilots  and  maintainers.  Communication  with  the  squadron  was  essential 
and  ongoing  throughout  the  research,  particularly  in  the  design  of  each  experiment.  Their 
input  narrowed  the  focus  of  the  research  and  helped  identify  the  most  important  aspects 
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of  the  problem  for  further  investigation.  They  were  also  important  in  validating  the 
results  from  the  perspective  of  the  final  customer. 

1.  Literature  Review 

There  is  almost  no  documentation  for  the  ITC  model  itself.  The  model  is 
proprietary  software  developed  for  Lockheed  Martin  and  not  meant  for  public  use.  The 
source  code  is  unavailable  and  any  change  to  the  model  must  be  made  through  the  Joint 
Program  Office  and  is  dependent  on  funding  availability.  Of  note,  the  model  has  been 
through  the  verification,  validation,  and  accreditation  process.  It  has  been  tested  on 
several  cases  using  the  F-16,  F-18,  AV-8B,  and  GR-7  aircraft  and  their  syllabuses.  Two 
prior  analyses  of  the  ITC  simulation  exist.  The  latest  was  completed  by  Joe  Dummar  for 
his  2011  master’s  thesis,  and  the  other  by  Marry  McDonald,  a  Research  Associate  at  the 
Naval  Postgraduate  School,  in  2010. 

Dummar’ s  thesis  shows  that  the  weather  model  used  by  the  ITC  is  wrong  and 
gives  inherently  optimistic  results.  The  ITC  uses  weather  for  all  aircraft  events.  Weather 
is  either  good  or  bad,  with  a  certain  probability  depending  on  the  month.  Each  aircraft 
event  has  an  independent  roll  of  the  dice  to  determine  if  weather  is  good  and  the  flight 
can  be  completed.  Dummar  suggests  changing  the  weather  modeling  in  the  ITC 
simulation  to  one  that  models  weather  using  Markov  chains  and  a  weather  resource,  since 
weather  conditions  for  consecutive  events  are  not  independent  (Dummar,  2011).  He 
finds  that  the  improved  approach  significantly  affects  estimates  of  TTT. 

McDonald’s  analysis  shows  the  performance  of  the  ITC  model  has  very  high 
variability  between  replication  (McDonald,  2010).  The  longest  delay  for  scheduled 
events  resulted  from  “student  match”  and  “night  flight.”  Student  match  means  a  student 
had  to  wait  for  another  student  in  order  to  complete  an  event  requiring  a  minimum  of  two 
students.  This  is  why  all  classes  in  the  ITC  are  started  in  multiples  of  two  students,  so 
they  can  progress  through  the  syllabus  together.  They  stay  synched  together  throughout 
their  experience  and,  if  one  falls  behind,  the  other  student  ends  up  waiting  longer  at  an 
event  queue  requiring  two  or  more  pilots.  This  is  a  simulation  limitation  that  can  lead  to 
longer  training  times  for  that  class,  but  is  an  issue  that  can  easily  be  resolved  at  the 
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squadron  level.  Night  flight  delay  means  a  student  has  run  out  of  time  to  complete  the 
night  event  before  the  ITC  closes  and  it  gets  rescheduled  for  the  next  day.  This  is  another 
simulation  limitation  that  may  lengthen  training  duration,  which  the  squadron  can  easily 
manage. 
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II.  DATA  AND  METHODOLOGY 


This  chapter  describes  the  ITC  model’s  behavior,  i.e.,  the  inputs,  outputs,  and 
ground  rules,  and  assumptions  required  to  initiate  a  run.  Descriptions  of  the  student  and 
resource  induction  fdes  are  also  covered. 

A.  METHODOLOGY 

The  ITC  model  simulates  day-to-day  ground  and  flight  operations  at  the  ITC.  The 
model  starts  collecting  data  when  students  and  resources,  such  as  aircraft  and  flight 
simulators  are  inducted  into  the  system  and  ends  when  the  last  student  graduates.  Each 
student  has  a  syllabus  that  lists  all  classes,  meetings,  and  required  activities  that  need  to 
be  finished  before  that  student  can  graduate.  It  also  lists  the  order  in  which  they  are 
taken.  The  model  strictly  enforces  this  order,  with  no  exceptions.  This  aspect  of  the 
simulation  is  a  departure  from  how  training  is  actually  completed  at  a  squadron.  This 
limitation  is  acknowledged  by  Lockheed  Martin,  who  claims  the  problem  is  “not  a  critical 
one  and  may  only  tend  to  make  the  ITC  throughput  estimates  conservative”  (Bom,  2007, 
p.  11).  Upon  graduation  of  the  last  student,  the  ITC  model  produces  detailed  reports  of 
each  student’s  experience.  The  time  they  start  and  finish  each  event  is  recorded,  as  well 
and  any  delay  and  reason  for  delay  of  any  syllabus  event.  Resource  utilization  rates  are 
recorded  and  used  as  justification  for  procurement  or  as  an  explanation  for  bottlenecks  in 
the  system. 

The  ITC  model  is  a  discrete-event,  first-in  first-out,  queuing-based  simulation.  It 
is  implemented  in  ModSim  III  version  2.1  (Kenney,  2009).  Currently,  all  random 
behavior  in  the  model  is  based  on  a  unifonn  zero  to  one  distribution.  Each  event  in  a 
syllabus  depends  on  wait  queues  that  holds  students,  instructors,  and  resources  needed  to 
complete  the  event.  Figure  2  shows  this  process  in  more  detail. 

Prior  to  a  flight  or  simulator  event,  the  students  must  prepare  for  the  event  at  the 
mission  planning  station  (MPS).  This  takes  a  fixed  period  of  time,  which  is  entered  via 
the  user  at  the  start  of  the  simulation  in  the  graphical  user  interface  (GUI).  Once 
planning  starts,  that  student  and  MPS  are  made  unavailable  until  completion  of  the 
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planning,  after  which  the  MPS  is  returned  to  the  MPS  queue  awaiting  another  user.  The 
student  then  waits  for  an  instructor  to  become  available.  Once  that  happens,  the 
instructor  is  taken  from  his  queue.  The  student  briefs  the  event  to  the  instructor  for  the 
time  specified  in  the  GUI;  then  both  wait  for  a  simulator  to  become  available.  Once  a 
simulator  is  available,  the  mission  is  flown  for  a  duration  dependent  on  the  student’s 
syllabus.  The  instructor  and  student  are  not  released  from  the  event  until  after  a  debrief, 
which,  again,  takes  a  fixed  period  of  time  and  is  defined  in  the  GUI.  At  that  point,  the 
instructor  is  returned  to  his  queue  and  the  student  will  move  to  his  next  event  in  the 
syllabus  with  a  probability  of  success  defined  in  the  GUI.  This  same  sequence  occurs  for 
each  simulator,  aircraft,  and  mission  trainer,  or  any  computer-based  training  requirement. 


3.  Full  Mission  Simulator  (FMS) 

Note:  MPS  =  Mission  Planning  Station  GUI  =  Graphical  User  Interface 


Note:  Re-fly  only  allowed 
Once  for  each  unique 
FMS  event 


•  2010  **•**»  Corporal** 


Figure  2.  Full  Mission  Simulator  (FMS)  Event  Flow  (From:  (Gray,  2010) 


The  ITC  model  is  designed  to  run  in  either  “steady  state”  or  “ramp  up”  mode. 
The  steady  state  mode  is  used  once  all  the  resources  have  arrived  at  the  ITC  and  their 
numbers  will  remain  relatively  constant  for  the  duration  of  the  simulation’s  time  horizon. 
This  mode  is  useful  in  predicting  the  impacts  of  maintenance  stand-downs,  long-term 
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runway  closures,  or  any  resource  constraint  that  affects  an  already  operational  training 
squadron.  The  ramp  up  mode  is  useful  in  predicting  the  impacts  of  different  resource 
arrival  schedules.  Depending  on  when  the  first  students  begin  to  arrive  for  training,  not 
all  the  resources  approved  may  be  ready  for  use  at  the  ITC.  The  resources  that  can  be 
varied  in  the  ramp  up  mode  are  the  number  of  FMSs,  training  aircraft,  and  instructor 
pilots.  These  inputs  are  provided  to  the  model  in  the  form  of  a  text  file  containing  a 

column  for  the  week  and  a  column  for  the  change  in  number  of  that  resource.  This  is  the 

mode  that  most  accurately  describes  the  ITC  as  it  is  stood  up  and  is  the  one  that  is  studied 
in  this  thesis. 

B.  MODEL  INPUT  AND  OUTPUT 

1.  Resources 

The  following  is  a  list  of  resources  accounted  for  in  the  ITC  model  (Kenney, 
2009,  p.  7): 

•  Training  Devices 

•  Classrooms 

•  Self-paced  Computer  Based  Training  Work  Stations 

•  Mission  Brief/De-brief  Rooms  for  completion  of  flight  events 

brief/debrief 

•  Military  Instructor  Pilots  (IPs) 

•  Military  Maintainer  Instructors 

•  Contract  academic  instructor  pilots 

•  Training  Aircraft 

•  Runways  and  training  airspace  (military  operating  areas,  low  level 

routes,  scored  target  ranges). 

Figure  3  shows  an  example  of  the  GUI.  On  this  page,  changes  can  be  made  to  the 
ITC  facility  resources,  the  air  space  resource,  and  the  weather  cancelation  rates.  The  user 
can  also  specify  the  location  of  the  pilot  syllabus  file  and  indicate  whether  to  induct 
aircraft  and  FMSs,  or  use  the  steady  state  number  shown  on  the  GUI. 
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F-35  Program  Information 

Non  Export  Controlled  Information  •  Releasable  to  Foreign  Persons 
Lockheed  Martin  Proprietary  Information 


Facility/  AncraH  Resources 

Student  Sylabus  Inducfions 

Unclassified  Classrooms 

fi¬ 

Pilot  Syllabus 

|  Multiple 

3 

Classified  Classrooms 

ts- 

Pilot  Inductions 

|USMC_COA4_E5hi.lxt 

“3 

ICW  Work  Stations 

[24 

Full  Mission  Simulator 

[5 

P  Induct  FMS 

r  Maint  Ski  Sets  >  3  (Single  Induction  Fie) 

MRT  Right  Simulator 

P* 

|  E  glin J  T  r  _M  2_M  S  S  P.  _S  ec  6_ 

200801  22  m  - 

Mission  Planning  Station 

(20 

Brief/Debrief  Rooms 

[45 

Avionics  Syllabus 

|Avioncs_lnitial_2  txt 

zi 

Aircraft  (Single  Seat) 

(l  50 

Avionics  Inductions 

(Zero_Avtorncs.txt 

“I! 

Aircraft  (Two  Seat) 

[5 

P  Induct  Aircraft 

ASMT 

[7 - 

Mechanical  Syllabus 

|  M  echamcalj  nitial_2  txt 

“31 

WLT 

[5— 

Mechanical  Inductions 

|Zero_M  echanrcal  bet 

- 3 

ESMT 

1^— 

Engine  Trainer 

F" 

Weapons  Syllabus 

(Weapons_lnitial_2  txt 

OML 

(6 

Weapons  Inductions 

|Zero_Weapons  txt 

“3 

Nole:  If  CFS  *  0,  will  use  FMS  Copyright  Lockheed  Marlin  Corporation.  October  2007 


Airspace  Runway  Availability 


MOA  Slots 

MOA/T aiget  Slot  holds  One  Mission 

Low  Levels 

Low  Level  Entry  Spacing  *  6  minutes  T arget  Slots 

Runways 


I®- 

[30— 

l»~ 

|5— 


Sortie  weather  cancellation  rale  f%) 

Oct -Feb  fl 5  Mar -Sept  (35 

Note:  Simulation  Starts  m  January 


OK _ |  Cancel  | 


Figure  3.  Integrated  Training  Center  Graphical  User  Interface  Page  One 


2.  Inputs 

This  is  the  collection  of  controlled  and  uncontrolled  variables  or  factors  to  be 
adjusted  in  the  design  of  experiments.  The  user  can  make  inputs  via  a  GUI  or  input  text 
file  (Kenney,  2009,  p.  8). 

•  Student  Induction  Schedule  (text  file). 

•  Course  Curriculum/Syllabus  (text  file). 

•  Training  Equipment  and  Aircraft  Resources  (GUI). 

•  Instructor  Resources  (GUI). 

•  Airspace  and  Runway  Resources  (GUI). 

•  Weather  Cancellation  Rates  (GUI). 

•  Full  Mission  Simulator  Planning,  Brief,  Debrief  duration  (GUI). 

•  Probability  of  Aircraft  Maintenance  (GUI). 

•  Aircraft  Repair  Time  (GUI). 
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•  Aircraft  Preparation  Time  for  the  Next  Flight  (GUI). 

•  Simulator  Training  Event  Re-fly  Rates  (GUI). 

•  Aircraft  Training  Event  Re-fly  Rates  (GUI). 

•  ITC  Training  Duration  Days/Year  (GUI). 

•  ITC  Training  Duration  Hours/Day  (GUI). 

Within  the  input  text  file  there  are  more  parameters  that  can  be  varied.  From  the 
student  induction  fde  the  number  of  students  per  class,  student  work  day,  and  student 
start  time  can  be  changed.  This  is  an  exhaustive  list  of  all  factors  the  user  can  adjust. 
Not  all  of  these  factors  are  changed  in  every  experiment. 

3.  Input  Text  Files 

All  of  the  experiments  will  use  essentially  the  same  induction  file.  Any 
modification  of  the  default  induction  file  for  a  particular  experiment  is  explained  in 
Chapter  III,  Design  of  Experiments. 

a.  Student  Induction  File 

The  student  induction  file  tells  the  simulation  what  week  a  new  class 
starts,  how  many  students  are  in  each  class,  what  syllabus  each  class  is  assigned,  the  time 
their  workday  starts,  and  how  long  their  workday  is.  The  default  induction  file  has 
60  classes,  with  two  students  starting  in  each  class.  The  first  students  start  in  week  83 
and  the  last  students  start  in  week  230,  a  period  of  almost  three  years.  The  students  all 
have  a  0700  start  time  and  a  9-hour  workday.  Each  class  is  assigned  one  of  14  different 
syllabuses.  Multiple  classes  can  be  inducted  on  the  same  week.  Table  1  is  an  example  of 
the  default  student  induction  schedule  for  the  first  18  classes.  The  actual  schedule  is  a 
text  file  and  does  not  contain  any  of  the  header  information. 
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Table  1.  Student  Induction  File 


Week 

Schedule 

Students 

Start  Hour 

Length  Day 

83 

DCMAJnitial_Cadre.txt 

10 

700 

9 

87 

B1.0_TX_ALL_B_Eglin.txt 

10 

700 

9 

93 

DCMAJnitial_Cadre.txt 

10 

700 

9 

93 

TX3_B10.txt 

10 

700 

9 

93 

TX4_B10.txt 

10 

700 

9 

98 

B1.0_TX3_A_Eglin.txt 

10 

700 

9 

98 

B1.0_TX4_A_Eglin.txt 

10 

700 

9 

104 

TX3_B10.txt 

10 

700 

9 

108 

B1.0_TX_ALL_B_Eglin.txt 

10 

700 

9 

111 

TX4_B10.txt 

10 

700 

9 

111 

TX4_B10.txt 

10 

700 

9 

111 

TX3_B10.txt 

10 

700 

9 

117 

B1.0_TX_ALL_B_Eglin.txt 

10 

700 

9 

120 

TX3_B10.txt 

10 

700 

9 

123 

TX3_B10.txt 

10 

700 

9 

123 

TX4_B10.txt 

10 

700 

9 

129 

TX4_B10.txt 

10 

700 

9 

b.  Aircraft  Induction  File 

This  file  tells  the  simulation  at  what  week  to  change  the  aircraft  inventory. 
Aircraft  are  added  or  removed  from  the  inventory  at  the  beginning  of  the  week.  There 
can  never  be  a  negative  inventory  and  there  is  no  maximum.  The  first  aircraft  are 
expected  to  arrive  in  week  83,  which  corresponds  to  October  2011,  and  the  last  aircraft  is 
to  arrive  in  September  2012.  After  September  2012,  or  approximately  week  127  of  the 
ITC  calendar,  there  will  be  a  total  of  12  aircraft  at  the  ITC.  This  schedule  is  based  on 
VMFAT-501’s  fiscal  year  2012  flight  hour  plan  and  is  subject  to  change.  For  simulation 
purposes,  the  exact  dates  are  not  required  as  long  as  each  experiment  uses  the  same 
induction  file. 


c.  Instructor  Pilot  Induction  File 

Instructor  pilots  are  inducted  at  a  slightly  faster  rate  than  the  aircraft. 
Some  of  the  instructors  are  removed  to  fill  instructor  roles  at  other  squadrons.  The  end 
goal  is  to  have  the  same  number  of  instructors  as  training  aircraft  (Kenney,  2009).  The 
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induction  file  contains  the  week  an  instructor  is  added  or  removed  from  the  ITC.  Any 
changes  take  place  at  the  beginning  of  the  week.  The  inventory  cannot  go  negative  and 
there  is  no  limit  as  to  how  many  instructors  are  in  the  ITC.  Unlike  other  resources, 
having  too  many  instructors  can  affect  the  TTT.  All  winged  aviators,  including  IPs,  are 
required  to  maintain  a  minimum  number  of  annual  flight  hours  for  proficiency 
(Operations,  2009).  Individual  squadrons  or  wings  may  further  restrict  the  requirements 
to  monthly  minimums.  Having  too  many  instructors  could  lead  to  a  loss  of  flight  hours 
available  for  the  students  and  increase  their  TTT.  This  is  because  there  are  limits  to 
cumulative  flight  hours  per  month  and  flight  hours  per  aircraft.  These  limits  are  not 
modeled  in  the  ITC  model.  However,  the  ITC  model  records  the  levels  used. 

d.  FMS  Induction  File 

Flight  simulators  are  an  essential  part  of  the  ITC  and  are  introduced  in  a 
similar  fashion  as  the  aircraft.  As  of  spring  2011,  VMFAT-501  is  expecting  four  FMSs 
(A.  C.  Libennan,  personal  communication,  April  18,  2011).  The  default  FMS  file  has 
four  FMSs  arriving  according  to  a  schedule  provided  by  VMFAT-501 ’s  training  officer. 

4.  Outputs 

After  each  run  of  the  simulation,  when  the  last  student  graduates,  the  following 
items  are  output  by  the  model  (Kenney,  2009,  p.  8): 

•  Student  Time-to-Train. 

•  Number  of  Students  in  training  as  a  function  of  time  (i.e.,  student 
on-board  count). 

•  Resource  Utilization  Hours  (classrooms,  training  devices, 
instructors,  aircraft,  ranges). 

•  Training  Equipment  Utilization  Rates  as  a  percentage  of  the 
training  day  for  each  month  of  the  simulation  run  (this  is  also 
shown  for  each  day  of  the  simulation  run  for  the  FMS). 

•  Aircraft  Flight  Hours/month. 

•  Aircraft  takeoff  and  landing  times,  including  any  delays 
experienced  for  the  runway,  airspace,  and  target  ranges. 

•  The  training  day  and  number  of  instances  on  that  day  that  a 

classroom  was  not  available  when  required. 
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•  Instructor  training  hours  per  week. 

The  output  from  all  runs  are  merged  to  get  a  distribution  of  outcomes,  which  are 
analyzed  to  find  important  trends  and  decisions  that  minimize  the  student’s  TTT  or 
maximize  a  training  resource.  Figure  4  summarizes  the  process  up  to  the  analysis: 


Figure  4.  Integrated  Training  Center  Overview  Diagram  (From:  Born,  2007) 


C.  GROUND  RULES  AND  ASSUMPTIONS 

A  basic  understanding  of  the  ground  rules  and  assumption  is  important  in 
understanding  how  the  model  attempts  to  imitate  reality  and  where  it  can  break  down.  It 
also  helps  in  the  design  of  experiments  and  how  to  interpret  the  results  of  the  model.  The 
following  assumptions  are  verbatim  from  the  “model  accreditation  support  package” 
(Kenney,  2009,  pp.  8-10). 

•  Normal  operating  hours  for  the  Integrated  Training  Center  are 
0700-2300  Monday  through  Friday.  Briefing  rooms  and 
equipment  will  be  available  to  support  a  0700  take  off  and 
commencement  of  simulator  events  at  0700.  Simulation  of  a 
single  shift  operation  can  be  accomplished  by  giving  all  students 
the  same  work  day  start  time  (i.e.,  0800)  with  a  work  day  length  of 
9  hours  (workday  ends  at  1700). 

•  Students  are  inducted  evenly  throughout  the  year  and  commence 
training  on  Monday  of  their  induction  week.  The  induction  files 
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can  be  adjusted  to  simulate  a  non-unifonn  flow  of  students  into 
the  ITC. 

More  than  one  class  may  be  inducted  in  the  same  week. 

The  optimum  pilot  class  size  for  JSF  is  six  students.  The  current 
ITC  classroom  design  for  pilots  can  accommodate  up  to  12 
students  per  class. 

Pilot  and  Maintainer  classes  are  service  unique. 

The  first  flight  brief  for  the  day  occurs  at  0445  for  a  0700  takeoff. 

The  first  flight  simulator  brief  of  the  day  occurs  at  0600  for  a  0700 
simulator  event. 

The  student  workday  is  a  9.0  hour  day  (start  to  finish)  and  students 
are  assigned  a  specific  “start  workday  time.”  The  intention  of  the 
9.0  hour  day  is  to  give  each  student  8.0  hours  per  day  when 
training  can  be  accomplished  and  1.0  hour  per  day  for  a  meal 
break.  Maintenance  training  for  the  U.S.  Services  is  accomplished 
on  two  shifts. 

In  no  case  will  students  and  instructors  be  scheduled  for  a  training 
event  sooner  than  12.0  hours  following  the  completion  of  their  last 
event  from  the  previous  day.  As  an  example,  a  student  and/or 
instructor  who  finishes  a  flight  debrief  at  2200  will  not  be 
scheduled  for  a  training  event  before  1000  the  next  day. 

Maximum  crew  duty  day  for  instructor  pilots  is  12.0  hours 
measured  from  the  start  of  the  first  training  event  to  engine 
shutdown.  This  ground  rule  is  based  on  a  USAF  Instruction  AFI 
1 1-202  Volume  3  General  Flight  Rules,  “For  single  seat  aircraft  or 
when  only  one  pilot  has  access  to  the  flight  controls,  the  maximum 
flight  duty  period  is  12  hours.”  The  normal  instructor  work  day  is 
9.0  hours. 

A  Full  Mission  Simulator  event  and  flight  event  will  be  pennitted 
on  the  same  day  for  the  same  student  if  they  can  both  be 
accomplished  within  the  students  9.0  hour  work  day. 

Flight  simulator  set  up  time  is  15  minutes  for  each  training  event 
(after  initial  daily  startup). 

Students  will  be  permitted  to  fly  more  than  one  flight  event  on  the 
same  day.  This  only  occurs  for  short  flights  such  as  Field  Carrier 
Landing  Practice  events  due  to  work  day  length  restrictions. 

Student  planning  time  for  each  flight  event  is  1.0  hour  utilizing  the 
Mission  Planning  Station. 

Brief  time  for  training  flights  is  2.25  hours  prior  to  takeoff. 


•  Flight  debriefs  end  3.0  hours  after  landing. 

•  Day  flights  land  before  1900  and  night  flights  takeoff  after  1900. 

•  Airborne  aggressor  requirements  are  satisfied  by  Training 
Squadron  JSF  aircraft  flown  by  Training  Squadron  instructor 
pilots. 

•  After  landing,  there  is  a  10%  probability  the  aircraft  will  require 
repair.  Time  to  repair  is  3.0  hours  followed  by  a  1.5  hour 
preparation  time  for  the  next  flight  event.  These  parameters  are 
adjustable  via  the  user  interface. 

•  Aircraft  turnaround  time  for  the  next  flight  event  is  1.5  hours  and 
is  adjustable  via  the  user  interface. 

•  Cancellations  for  weather  occur  after  student  planning  and  prior  to 
commencement  of  the  flight  brief.  Weather  cancellation  rates  are 
5%  and  are  adjustable  via  the  user  interface. 

•  Re-fly  rates  for  flights  and  simulator  events  are  8%  and  5% 
respectively  and  are  adjustable  via  the  user  interface.  The  re-fly 
decision  is  made  at  the  end  of  debrief. 

•  Navy  Carrier  Qualification  (CQ)  re-fly  rate  is  12.5%.  Students 
who  repeat  CQ  will  re-fly  ah  simulator  and  flight  events  associated 
with  the  CQ  phase  of  training. 

•  Students  will  not  re-fly  the  same  flight  or  simulator  event  more 
than  once. 

•  Student  planning  time  for  each  Full  Mission  Simulator  event  is  30 
minutes  and  is  adjustable  via  the  user  interface.  Student  planning 
utilizes  a  Mission  Planning  Station. 

•  Full  Mission  Simulator  brief  time  is  60  minutes  and  is  adjustable 
via  the  user  interface. 

•  Full  Mission  Simulator  debrief  time  is  30  minutes  and  is  adjustable 
via  the  user  interface. 

•  The  number  of  Military  instructor  pilots  is  equal  to  the  number  of 
training  aircraft  in  accordance  with  direction  received  from  the 
Services  and  the  JSF  Program  Office.  Ah  airborne  flight 
instruction  is  provided  by  Military  instructor  pilots. 

•  After  completing  a  flight  debrief,  the  instructor  will  not  be 
scheduled  for  a  subsequent  flight  brief  for  1 .0  hour. 

•  On  each  training  day,  there  is  a  2%  probability  that  each  instructor 
and  student  may  be  unavailable  for  a  2.0  hour  time  period.  This  is 
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intended  to  account  for  medical  appointments  and  unexpected 
absences  for  both  instructors  and  students. 

•  Minimum  allowable  spacing  between  flights  on  the  same  low-level 
route  is  10  minutes. 

•  Landing  aircraft  have  priority  over  departing  aircraft  for  the 
runway. 

•  There  are  a  total  of  246  days  per  year  that  are  available  for 
[training].  This  parameter  is  selectable  via  the  user  interface. 

•  Students  will  take  a  15-minute  break  in  between  non-like  training 
events  (e.g.  simulator  to  classroom,  classroom  to  simulator,  etc.). 

Breaks  during  successive  classroom  training  events  will  be  handled 
by  the  instructor. 

The  ITC  model  repeatedly  uses  a  point  estimate  rather  than  a  random  draw  from  a 
distribution  for  its  events.  For  some  events,  such  as  maintenance  repair  times,  subject 
matter  experts  expect  significant  variability.  Using  point  estimates  in  lieu  of  probability 
distributions,  especially  in  high-utilization  situations,  will  likely  result  in  overestimating 
the  system’s  throughput. 

These  model  assumptions  are  used  in  conjunction  with  recommendations  from 
VMFAT-501  to  help  design  and  analyze  the  experiment.  Their  contribution  cannot  be 
summarized  in  table  form,  but  is  an  important  part  of  how  the  simulations  are  designed 
and  what  factors  need  to  be  investigated.  This  helps  reduce  the  complexity  of  the  data 
and  reduce  the  number  of  COAs  to  explore. 

In  Chapter  III,  these  factors  and  assumptions  are  brought  together  to  form  an 
efficient  design  that  can  be  used  to  get  the  most  information  from  the  model,  while 
keeping  within  computational  and  time  constraints.  Emphasis  is  on  what  factors,  at  what 
levels,  have  the  greatest  effect  on  TTT  and  how  VMFAT-501  can  use  that  information  to 
mitigate  any  unforeseen  circumstances. 
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III.  DESIGN  OF  EXPERIMENTS 


This  chapter  discusses  how  the  experiments  in  this  thesis  are  designed  and  the 
reason  behind  those  designs.  There  are  three  separate  experiments  intended  to  answer  the 
questions  posed  in  Chapter  I.  The  first  experiment  analyzes  the  effect  of  aircraft 
upgrades  and  modifications.  The  second  experiment  analyzes  airspace  and  its  effect  on 
the  ITC  model.  The  last  experiment  analyzes  syllabus  variables — those  factors  that  can 
be  changed  in  the  student  induction  files — and  their  effects  on  TTT.  The  noise  factors, 
signal  factors,  and  the  levels  of  each  in  the  three  separate  experiments  are  explained  in 
detail. 

The  designs  are  run  on  a  cluster  of  computers  at  the  Naval  Postgraduate  School’s 
Simulation  Experiments  and  Efficient  Designs  (SEED)  Center  in  Monterey,  California. 
Each  design  point  is  repeated  50  times  with  different  random  number  seed  values.  The 
outputs  of  each  run  are  concatenated  together  to  fonn  output  files  representing  all  the 
runs  and  repetitions.  This  output  is  merged  with  the  original  design  and  analyzed  using 
the  JMP  statistical  package  (Sail,  Creighton,  &  Lehman,  2005). 

A.  WHY  DESIGN  EXPERIMENTS? 

There  are  many  different  factors  and  levels  of  those  factors  in  the  ITC  model  to 
consider,  and  only  a  limited  amount  of  computer  resources  and  time  to  complete  the 
study,  thus  an  efficient  design  of  experiments  is  crucial.  With  a  proper  design,  all  aspects 
of  a  complex  model  can  be  studied  with  greater  fidelity  and  exponentially  less  time  than 
looking  at  all  combinations  of  factors  and  levels.  The  majority  of  the  experiments  use  a 
mixed  design,  based  on  crossing  a  traditional  factorial  design  with  a  nearly  orthogonal 
Latin  hypercube  (NOLH)  design  (Cioppa  &  Lucas,  2007).  The  NOLH  is  the  basis  for  all 
continuous  factors  of  concern,  while  categorical  factors  use  the  factorial  design.  The 
algorithm  used  for  their  generation  can  be  found  on  the  SEED  Center  website 
(http://harvest.nps.edu/). 
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1. 


Robustness 


The  goal  sometimes  is  not  just  finding  a  solution  that  maximizes  or  minimizes  the 
metric  of  interest,  but  a  solution  that  also  looks  at  the  variability  of  that  optimal  solution. 
A  robust  solution  is  one  that  may  sacrifice  mean  performance  to  minimize  variability.  It 
is  a  solution  that  is  stable  around  a  larger  variation  of  uncontrollable  inputs  with  little 
change  in  output.  This  way  of  looking  at  and  analyzing  the  data  is  useful  when  there  is  a 
lot  of  variability  in  the  response,  uncertainty  in  the  inputs,  and  when  the  variance  of  the 
response  is  not  constant  for  the  entire  range  of  the  regressors.  This  condition  applies  to 
our  situation. 

The  inputs  consist  of  signal  and  noise  factors.  The  signal  factors  are  inputs  that 
the  user  has  control  over.  The  number  of  aircraft,  instructor  working  hours,  and  students 
per  class  are  all  signal  factors.  The  weather’s  cancellation  rate,  aircraft  repair  times,  and 
aircraft  failure  rates  are  all  unknown  and  uncontrollable.  They  can  best  be  described  by  a 
distribution,  but  the  ITC  simulation  does  not  currently  take  probability  distributions  as 
input  variables.  Because  the  JSF  is  a  new  system,  the  distribution  of  failure  rates  and 
repair  times  are  unavailable.  The  model  uses  point  estimates  for  these  and  the  other  noise 
factors.  Unfortunately,  queuing  theory  tells  us  that  this  will  likely  bias  the  results 
(Ross,  2007). 

In  robust  analysis,  the  intent  is  to  look  at  the  effect  of  the  signal  factors  over  the 
entire  distribution  of  noise  factors.  To  do  this,  both  signal  and  noise  factors  are  expanded 
into  separate  orthogonal  or  nearly  orthogonal  designs  covering  the  full  range  of  their 
values.  These  two  separate  designs  are  then  crossed  to  form  a  master  design.  Once  the 
data  are  generated,  it  is  collapsed  over  the  noise  variables.  That  is,  for  a  given  signal 
design  point,  the  response  is  generated  by  averaging  all  of  the  design  points  in  the  noise 
matrix.  This  gives  a  distribution  of  outcomes  for  each  design  point  of  the  signal  factors. 
This  is  better  than  using  a  single  value  for  the  noise  factors,  but  not  as  good  as  using  a 
distribution.  It  gives  equal  weight  to  all  possible  values  of  the  noise  factor  when  they  are 
probably  not  all  equal,  and  can  lead  to  higher  variability  in  the  outcome.  The 
methodology  used  for  the  robust  analysis  in  this  thesis  is  further  described  in 
Sanchez  (2000). 


20 


2. 


Factors  and  Levels 


The  factors  of  concern,  affecting  TTT  and  their  levels  (or  range  of  values),  are 
dependent  on  the  problem  being  studied.  All  factors  come  from  the  list  of  inputs  shown 
in  Chapter  II  or  otherwise  are  explained  separately.  Not  all  of  these  factors  are  used  in 
every  experiment,  nor  will  their  ranges  be  the  same.  Each  experiment  has  a  specific  goal, 
and  thus  the  range  and  number  of  factors  is  adjusted  to  best  match  the  need  of  the 
experiment.  If  the  full  values  of  each  factors’  ranges  were  used  they  may  dominate  the 
factors  of  interest  and  hide  the  response.  This  is  a  very  important  consideration  when 
varying  noise  factors  in  the  design,  especially  when  all  possible  levels  have  a  probability 
of  occurrence.  For  example,  having  aircraft  cancellation  rates  near  100%  will  dominate 
the  output  and  make  it  difficult  to  see  the  effects  of  the  signal  factors.  This  narrowing 
down  of  the  correct  range  for  the  each  factor  is  an  iterative  process  involving  subject 
matter  experts. 

Unless  otherwise  stated  in  the  experiment,  the  noise  factors  are  at  the  default 
values  used  in  historical  analysis,  shown  in  Table  2. 


Table  2.  Noise  Factors  Default  Levels 


Factor 

Level 

Factor 

Level 

MOA  Slots 

30 

FMS  Refly  % 

2 

Low  Level 

30 

Weather  cnx  Summer  % 

30 

Target  Slots 

30 

Weather  cnx  Winter  % 

15 

Runways 

5 

AC  Fail  Rate  % 

50 

Flight  Refly  % 

15 

AC  Repair  Time  (hours) 

3 

B.  AIRCRAFT  UPGRADE  AND  MODIFICATION  DESIGN  MATRIX 

Once  aircraft  are  delivered  to  the  ITC  they  will  be  basic  aircraft  with  minimal 
flight  clearances  and  not  yet  fully  mission  capable.  As  testing  continues,  and  the  aircraft 
are  certified  for  different  flight  profiles,  they  will  be  upgraded  at  the  ITC.  This  upgrade 
may  be  a  simple  software  replacement  or  a  complex  process  where  a  flight  control 
surface,  hydraulic,  electrical,  or  structural  component  is  replaced.  When  these  aircraft 
undergo  an  upgrade,  they  are  unusable  for  the  period  of  time  required  to  complete  the 
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upgrade.  A  question  of  interest  to  VMFAT-501  is  what  affect  the  upgrade  process  will 
have  on  TTT? 

Aircraft  module  upgrade  is  not  an  input  to  the  system.  The  only  way  to  simulate 
this  as  a  factor  is  by  using  a  perturbation  of  the  original  aircraft  induction  file.  The 
aircraft  delivery  schedule  is  known.  By  modifying  when  aircraft  arrive  and  taking  them 
out  of  service  when  they  undergo  upgrades,  multiple  scenarios  can  be  explored  and 
compared  to  a  base  case  of  no  upgrades.  Depending  on  the  complexity  of  the  upgrade, 
the  aircraft  downtime  could  be  from  a  few  days  to  a  few  weeks.  The  current  plan  is  to 
keep  four  aircraft  available  for  training  at  all  times,  while  the  other  aircraft  go  through  the 
upgrade  process  (A.  C.  Liberman,  personal  communication,  April  18,  2011).  With  this 
COA,  no  upgrade  is  made  until  there  are  more  than  four  aircraft  in  the  inventory. 

This  perturbation  of  the  aircraft  induction  schedule  is  based  on  the  assumption 
that  once  the  upgrade  is  made  available,  then  all  available  aircraft — less  those  needed  to 
complete  training  missions — will  all  undergo  the  upgrade  at  the  same  time  and  complete 
at  the  same  time.  The  time  needed  to  complete  the  upgrade  depends  on  its  complexity, 
but  is  assumed  to  be  between  one  and  three  weeks  for  simulation  purposes.  If  the 
upgrade  is  made  available  while  there  are  only  four  aircraft  in  the  inventory,  none  of 
those  aircraft  are  upgraded  until  a  new  aircraft  arrives.  The  upgrades  are  made  as  soon  as 
the  required  minimum  numbers  of  aircraft  are  present. 

The  idea  is  to  ensure  a  minimum  of  two  aircraft  at  all  times  to  cover  the  flight 

schedule,  while  the  extra  two  are  used  as  backups  for  any  maintenance  issues  that  may 

come  up.  This  strategy  of  four  to  make  two  may  not  be  the  best  choice  for  all  situations. 

Changes  in  upgrade  completion  time  or  aircraft  failure  rates  may  favor  a  different  COA. 

To  understand  which  COA  is  best,  the  following  rules  are  used  to  simulate  all  possible 

decision  policies  with  respect  to  how  many  aircraft  to  hold  in  reserve.  The  number  of 

aircraft  held  in  reserve  is  between  two  and  five,  and  is  designated  A,  B,  C,  or  D, 

respectively.  The  time  to  complete  is  one,  two,  or  three  weeks.  Although  it  may  take  a 

few  days,  the  simulation  only  accepts  integer  values  representing  the  week  to  make  a 

change  to  the  aircraft  inventory.  Added  to  this  is  a  control  design,  designated  AO,  where 

upgrades  are  not  required  or  they  take  zero  time  to  complete,  corresponding  to  the 
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original  aircraft  induction  schedule.  This  makes  a  total  of  13  different  aircraft  induction 
schedules  or  design  points  to  explore.  This  design  is  crossed  with  a  10  factor  (noise) 
NOLH  design  of  33  design  points,  forming  a  mixed  design  with  a  total  of  429  runs  to 
fully  explore  the  problem.  The  individual  upgrade  scenarios  are  shown  in  Tables  3  and  4. 
The  value  under  the  “week”  column  represents  when  the  aircraft  inventory  will  change 
and  the  value  under  “amount”  column  represents  the  number  of  aircraft  added  to 
the  inventory. 

Table  3.  Upgrade  Design  Signal  Factors 


Design 

Aircraft  to  Keep 

Upgrade  Time 

Control 

4 

0 

A1 

2 

1 

A2 

2 

2 

A3 

2 

3 

B1 

3 

1 

B2 

3 

2 

B3 

3 

3 

Cl 

4 

1 

C2 

4 

2 

C3 

4 

3 

D1 

5 

1 

D2 

5 

2 

D3 

5 

3 

Table  4.  Upgrade  Design  Schedule  Induction  File 


weekl  week2 

week3 

week4 

week5 

week6 

week7 

week8 

week9 

amountl 

amount2 

amount3  amount4  amount5  amount6 

amount7 

amount8 

amount9 

ACO 

83 

87 

90 

98 

118 

121 

126 

127 

128 

3 

2 

2 

1 

1 

2 

1 

0 

0 

AC1 

83 

86 

88 

91 

99 

119 

122 

127 

129 

2 

1 

2 

2 

1 

1 

2 

1 

0 

AC2 

83 

89 

92 

100 

120 

123 

128 

129 

130 

2 

3 

2 

1 

1 

2 

1 

0 

0 

AC3 

83 

87 

89 

90 

92 

101 

121 

124 

129 

2 

1 

1 

1 

2 

1 

1 

2 

1 

B1 

83 

89 

90 

91 

99 

119 

122 

127 

128 

3 

1 

1 

2 

1 

1 

2 

1 

0 

B2 

83 

90 

91 

92 

100 

118 

120 

123 

128 

3 

2 

1 

1 

1 

0 

1 

2 

1 

B3 

83 

93 

96 

101 

121 

124 

129 

130 

131 

3 

3 

1 

1 

1 

2 

1 

0 

0 

Cl 

83 

87 

91 

92 

99 

119 

122 

127 

128 

3 

1 

2 

1 

1 

1 

2 

1 

0 

C2 

83 

87 

91 

94 

100 

120 

123 

128 

129 

3 

1 

1 

2 

1 

1 

2 

1 

0 

C3 

83 

87 

96 

101 

121 

124 

129 

130 

131 

3 

1 

3 

1 

1 

2 

1 

0 

0 

D1 

83 

87 

93 

94 

99 

119 

122 

127 

128 

3 

2 

1 

1 

1 

1 

2 

1 

0 

D2 

83 

87 

96 

98 

100 

120 

123 

128 

129 

3 

2 

1 

1 

1 

1 

2 

1 

0 

D3 

83 

87 

98 

99 

121 

124 

129 

130 

131 

3 

2 

1 

2 

1 

2 

1 

0 

0 

The  noise  factors  used  in  this  experiment  are  shown  in  Table  5.  The  minimum 


and  maximum  value  of  each  factor  is  also  shown.  The  refly  values  represent  the 
probability  that  a  student  will  have  to  refly  an  aircraft  or  FMS  event.  “Wx  Summer”  and 
“Wx  Winter”  represent  weather  cancellation  rates  for  each  flight  in  the  summer  or  winter, 
respectively.  “Fail  Rate”  and  “Repair  Time”  represent  the  probability  that  an  aircraft  will 
break  down  before  the  flight  and  how  long  it  will  take  to  repair  the  aircraft. 
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Table  5.  Upgrade  Noise  Factors  Nearly  Orthogonal  Latin  Hypercube 


Min 

1 

1 

1 

1 

5 

5 

5 

5 

5 

1 

Max 

5 

33 

15 

15 

20 

20 

25 

15 

25 

6 

run 

Runways 

MOA  Slots 

Low  Level 

Target  Slots 

Flight  Refly 

FMS  Refly 

Wx  Summer 

Wx  Winter 

AC  Fail  Rate 

AC  RepairTime 

i 

1 

15 

4 

13 

14 

15 

14 

15 

19 

4 

2 

5 

5 

6 

8 

8 

16 

11 

14 

14 

5 

3 

3 

30 

3 

1 

14 

16 

6 

8 

24 

3 

4 

5 

33 

7 

14 

7 

17 

6 

9 

7 

2 

5 

1 

16 

4 

11 

16 

11 

16 

6 

8 

4 

6 

5 

11 

5 

7 

8 

7 

23 

5 

16 

5 

7 

3 

32 

5 

1 

15 

11 

23 

14 

5 

2 

8 

4 

31 

6 

14 

9 

8 

25 

10 

24 

2 

9 

2 

8 

8 

11 

10 

5 

9 

11 

17 

1 

10 

4 

10 

11 

4 

13 

6 

13 

13 

9 

1 

11 

2 

25 

15 

6 

6 

7 

8 

9 

18 

4 

12 

4 

22 

14 

12 

20 

12 

13 

7 

8 

4 

13 

2 

7 

9 

9 

7 

20 

21 

8 

11 

1 

14 

3 

13 

13 

3 

13 

19 

19 

8 

21 

2 

15 

2 

28 

14 

6 

5 

15 

20 

12 

12 

6 

16 

4 

20 

15 

12 

19 

13 

18 

13 

20 

4 

17 

3 

17 

8 

8 

13 

13 

15 

10 

15 

4 

18 

5 

19 

12 

3 

11 

10 

16 

5 

11 

3 

19 

1 

29 

10 

8 

17 

9 

19 

6 

16 

2 

20 

3 

4 

13 

15 

11 

9 

24 

12 

6 

4 

21 

2 

1 

9 

2 

18 

8 

24 

11 

23 

5 

22 

5 

18 

12 

5 

9 

14 

14 

14 

23 

3 

23 

1 

23 

11 

9 

17 

18 

8 

15 

14 

2 

24 

3 

2 

12 

15 

10 

14 

7 

6 

25 

5 

25 

2 

3 

10 

2 

16 

17 

5 

10 

6 

5 

26 

4 

26 

8 

5 

15 

20 

21 

9 

13 

6 

27 

2 

24 

5 

12 

12 

19 

18 

7 

21 

6 

28 

4 

9 

1 

10 

19 

18 

22 

11 

13 

3 

29 

2 

12 

2 

5 

5 

13 

17 

13 

22 

3 

30 

4 

27 

7 

7 

18 

5 

9 

12 

19 

6 

31 

3 

21 

3 

13 

12 

6 

11 

13 

9 

5 

32 

4 

6 

2 

10 

20 

10 

10 

8 

18 

1 

33 

3 

14 

1 

4 

6 

12 

12 

7 

10 

3 

The  correlation  of  this  design  and  its  space-filling  property  are  shown  in  Figure  5. 
Correlation  is  a  measure  of  linear  dependence  between  two  factors  and  is  a  value  between 
1  and  -1.  A  zero  correlation  would  mean  the  two  factors  have  no  linear  dependence  and 
is  the  goal  in  any  designing.  Correlation  of  1  or  -1  means  a  perfect  positive  or  negative 
linear  relationship  between  the  two  factors.  The  design  is  nearly  orthogonal,  with  “FMS 
Refly”  and  “Runways”  having  the  highest  correlation  of  only  8%.  This  should  not 
interfere  with  estimating  the  effects  of  different  upgrade  strategies.  The  scatterplot 
matrix  shows  the  coverage  achieved  in  the  factor  space  with  this  design.  Each  point 
represents  a  specific  run  of  the  experiment.  All  pairs  of  signal  and  noise  factors  are 
plotted. 
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Multivariate 

Correlations 


Runways  MOA  Slots  Low  Level  Target  Slots  Flight  Refly  FMS  Refly  Wx  Summer  Wx  Winter  AC  Fail  RateAG  Repair  Time 


Runways 

1.0000 

0.0381 

0.0111 

-0.0176 

-0.0694 

-0.0800 

0.0199 

-0.0654 

-0.0331 

0.0150 

MOA  Slots 

0.0381 

1.0000 

0.0219 

-0.0310 

-0.0154 

0.0196 

-0.0038 

0.0043 

-0.0241 

0.0163 

Low  Level 

0.0111 

0.0219 

1.0000 

-0.0018 

-0,0087 

-0.0151 

-0.0001 

0.0120 

-0.0158 

0.0761 

Target  Slots 

-0,0176 

-0.0310 

-0,0018 

1.0000 

0.0103 

-0.0500 

-0.0231 

-0,0049 

0.0048 

0.0345 

Flight  Refly 

-0.0694 

-0.0154 

-0.0087 

0.0103 

1.0000 

-0.0173 

0.0056 

0.0347 

-0.0360 

0.0096 

FMS  Refly 

-0.0800 

0.0196 

-0.0151 

-0.0500 

-0,0173 

1.0000 

-0.0057 

-0.0101 

-0,0000 

0.0695 

Wx  Summer 

0.0199 

-0,0038 

-0,0001 

-0.0231 

0.0056 

-0,0057 

1.0000 

-0.0241 

0.0008 

0.0392 

Wx  Winter 

-0.0654 

0.0043 

0.0120 

-0.0049 

0.0347 

-0.0101 

-0.0241 

1.0000 

-0.0155 

-0.0164 

AC  Fail  Rate 

-0.0331 

-0.0241 

-0.0158 

0.0048 

-0.0360 

-0.0000 

0.0008 

-0.0155 

1.0000 

0.0719 

AC  Repair  Time 

0.0150 

0.0163 

0.0761 

0.0345 

0.0096 

0.0695 

0.0392 

-0.0164 

0.0719 

1.0000 
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Figure  5.  Upgrade  Design  Multivariate  Plot  of  Noise  Factors 
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C.  AIRSPACE  EFFECTS  ON  TTT 

This  experiment  focuses  on  limiting  the  number  of  Military  Operating  Areas 
(MOAs),  low-level  routes,  and  target  slots  available  for  training.  Each  syllabus  flight 
event  uses  one  or  more  of  these  resources  to  complete  its  mission.  An  MOA  is  airspace 
reserved  for  military  training  and  test  flights.  Low-level  routes  are  routes  in  an  MOA,  or 
other  special-use  airspace,  where  military  aircraft  can  train  at  speeds  and  altitudes  that  are 
otherwise  prohibited  under  Federal  Aviation  Administration  rules.  Target  slots  or  ranges 
are  usually  located  in  restricted  airspace  and  are  used  by  the  military  for  live-fire 
exercises.  All  of  these  areas  are  shared  with  other  military  organizations  and  scheduled 
through  a  local  control  authority.  The  availability  of  these  resources  could  then  affect 
TTT  in  their  own  way.  The  ITC  has  queues  for  each  of  the  airspace  factors  and 
depending  on  when  and  how  many  students  need  these  resources  there  may  be  a  delay  in 
its  availability.  For  this  experiment,  the  noise  and  signal  factors,  and  their  levels,  are 
shown  in  Table  6. 


Table  6.  Airspace  Experiment  Input  Variables 


SIGNAL 

NOISE 

Min 

1 

1 

1 

1 

5 

0 

5 

0 

5 

1 

Max 

30 

30 

30 

5 

20 

20 

25 

15 

25 

6 

MOA  Slots 

Low  Level 

Target  Slots 

Runways 

Flight  Refly 

FMS  Refly 

Wx  Summer 

Wx  Winter 

AC  Fail  Rate 

AC  Repair  Time 

10 

30 

25 

2 

20 

16 

13 

4 

24 

4 

3 

8 

26 

1 

9 

18 

16 

0 

11 

4 

5 

14 

3 

2 

12 

1 

10 

9 

21 

6 

6 

19 

10 

2 

14 

6 

25 

8 

8 

5 

23 

28 

14 

4 

19 

9 

8 

5 

5 

5 

30 

10 

12 

5 

10 

8 

21 

1 

20 

5 

19 

6 

30 

4 

8 

20 

11 

13 

14 

6 

17 

26 

23 

3 

18 

15 

24 

12 

18 

4 

16 

16 

16 

3 

13 

10 

15 

8 

15 

4 

21 

1 

6 

4 

5 

4 

18 

11 

6 

3 

28 

23 

5 

5 

16 

3 

14 

15 

19 

3 

26 

17 

28 

5 

13 

19 

20 

6 

9 

1 

25 

12 

21 

4 

11 

14 

5 

7 

23 

2 

8 

3 

17 

2 

6 

11 

23 

10 

25 

2 

1 

21 

19 

1 

15 

13 

9 

14 

10 

2 

12 

25 

1 

3 

17 

0 

19 

2 

16 

1 

14 

5 

8 

3 

7 

5 

6 

3 

13 

3 

12 

8 

28 

5 

17 

8 

10 

14 

16 

3 

17 

1 

10 

2 

18 

11 

5 

5 

18 

1 

8 

19 

25 

3 

6 

5 

18 

12 

25 

2 

30 

17 

5 

4 

10 

20 

16 

2 

20 

2 

5 

10 

1 

5 

12 

3 

11 

0 

21 

5 

25 

3 

23 

2 

11 

16 

6 

11 

23 

6 

10 

26 

14 

2 

20 

6 

23 

7 

24 

4 

28 

25 

19 

5 

16 

19 

21 

9 

19 

4 

19 

23 

3 

1 

8 

13 

20 

1 

14 

4 

14 

30 

21 

4 

7 

9 

25 

10 

13 

6 

23 

12 

6 

3 

19 

15 

13 

3 

5 

5 

1 

14 

26 

3 

15 

0 

14 

13 

10 

5 

26 

21 

30 

1 

13 

18 

19 

15 

9 

2 

6 

28 

8 

4 

14 

4 

24 

4 

8 

1 
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5 

17 

4 

5 

14 

8 

8 

6 

3 

3 

6 

12 

2 

9 

1 

9 

6 

11 

3 
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Each  design  is  created  using  an  NOLH.  The  final  design  is  a  cross  of  the  two 
separate  designs  to  form  a  composite  design  with  1,089  design  points,  which  allows  us  to 
examine  the  relationship  between  airspace  availability  and  TTT.  The  multivariate 
correlation  plot  of  the  final  design  is  shown  in  Figure  6.  The  highest  correlation  between 
the  signal  factors  is  a  very  low  0.0053. 


Correlations 

MOA.SIots 

Low.  Level  Target.Slots 

Runways  Flight.  Refly  FMS.  Refly  Wx.Summer  Wx.  Winter  AC.  Fail.  Rate  AC.Repair.Time 

MOA.SIots 

1.0000 

-0.0051 

0.0001 

0.0000 

-0.0000 

-0.0000 

-0.0000 

0.0000 

-0.0000 

-0.0000 

Low.  Level 

-0.0051 

1.0000 

0.0053 

-0.0000 

-0.0000 

0.0000 

-0.0000 

-0.0000 

-0.0000 

0.0000 

Target.Slots 

0.0001 

0.0053 

1.0000 

0.0000 

0.0000 

-0.0000 

-0.0000 

-0.0000 

-0.0000 

0.0000 

Runways 

0.0000 

-0.0000 

0.0000 

1.0000 

-0.0157 

-0.0691 

0.0244 

-0.0003 

0.0019 

0.0141 

Flight.  Refly 

-0.0000 

-0.0000 

0.0000 

-0.0157 

1.0000 

-0.0054 

-0.0097 

0.0004 

0.0341 

-0.0075 

FMS.  Refly 

-0.0000 

0.0000 

-0.0000 

-0.0691 

-0.0054 

1.0000 

-0.0081 

-0.0001 

-0.0004 

-0.0126 

Wx.Summer 

-0.0000 

-0.0000 

-0.0000 

0.0244 

-0.0097 

-0.0081 

1.0000 

-0.0022 

-0.0004 

-0.0344 

Wx.  Winter 

0.0000 

-0.0000 

-0.0000 

-0.0003 

0.0004 

-0.0001 

-0.0022 

1.0000 

0.0053 

0.0353 

AC.  Fail.  Rate 

-0.0000 

-0.0000 

-0.0000 

0.0019 

0.0341 

-0.0004 

-0.0004 

0.0053 

1.0000 

0.0154 

AC.Repair.Time 

-0.0000 

0.0000 

0.0000 

0.0141 

-0.0075 

-0.0126 

-0.0344 

0.0353 

0.0154 

1.0000 

Figure  6.  Airspace  Design  Multivariate  Plot 


D.  SYLLABUS  VARIABLES 

This  last  experiment  tests  the  effect  of  changing  student  start  time,  workday 
length,  number  of  FMSs,  and  ITC  work  hours  on  TTT.  The  first  two  experiments  vary 
each  signal  factor  over  the  entire  range  of  noise  factors,  but  for  this  experiment,  the  noise 
factors  are  kept  constant  at  their  default  values.  By  looking  at  only  the  signal  factor,  the 
degree  of  complexity  of  the  design  is  reduced  and  allows  a  full-factorial  design  to  be 
used.  The  signal  factors  are  listed  in  Table  7,  with  information  on  their  levels  and 
variable  types. 


Table  7.  Syllabus  Signal  Factors 


Factor 

Type 

Levels 

Number  of  students  in  class 

Integer 

2,6,  10 

Number  of  FMSs 

Integer 

4,5,6 

Start  time 

Continuous  (hrs) 

0600,  0900 

Length  of  workday 

Continuous  (hrs) 

8,  11 

Schoolhouse  hours 

Continuous  (hrs) 

12,  19 

Increasing  the  class  size  directly  affects  TTT.  As  more  students  complete  for  a 
constant  supply  of  resources,  the  availability  of  these  resources  become  limiting  and 
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bottlenecks  are  created.  This  results  in  more  time  spent  waiting  in  queues  and  increases 
TTT.  This  factor  was  added  to  get  a  sense  of  how  much  that  increase  is  and  how  the 
other  signal  variables  affect  TTT  as  student  numbers  increase.  The  students  have  a 
default  start  time  and  workday  length  of  0700  and  nine  hours,  respectively.  Increasing  or 
decreasing  the  student  start  time  allows  more  day  or  night  flight  events  to  be  completed. 
Changing  the  workday  length  allows  a  student  more  time  to  schedule  and  complete 
events  each  day.  Increasing  schoolhouse  hours  allows  more  time  for  simulator  and 
academic  events  to  be  scheduled  in  a  single  day.  Increasing  the  number  of  FMSs  should 
result  in  a  decrease  in  the  average  TTT ;  but  at  what  level  of  class  size  will  this  have  the 
most  pronounced  effect? 

The  goals  for  this  experiment  are  screening  the  factors  of  interest,  then  learning 
what  levels  of  those  factors  will  minimize  TTT.  A  full-factorial  design  is  created  with 
JPM  software  (Sail,  Creighton,  &  Lehman,  2005).  From  the  data,  a  meta-model  of  the 
system  is  created.  Both  linear  regression  and  partition  trees  are  used  to  explore  the  data. 
This  design  enables  main  effects  and  two-way  interaction  to  be  estimated  with  no 
confounding.  Table  8  shows  the  design  in  tabular  form. 
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Table  8.  Full-Factorial  Design 


Design  Pt 

Start  Time 

Stud  Workday 

ITC  Workday 

Num  in  Class 

Num  FMS 

1 

6 

11 

12 

io 

6 

2 

9 

8 

19 

6 

5 

3 

6 

11 

19 

6 

6 

4 

9 

8 

12 

2 

5 

5 

9 

8 

19 

IO 

5 

6 

9 

11 

19 

6 

5 

7 

9 

8 

19 

2 

6 

8 

6 

8 

19 

2 

6 

9 

6 

11 

12 

2 

5 

10 

9 

11 

12 

6 

4 

11 

9 

8 

12 

IO 

4 

12 

6 

8 

19 

6 

4 

13 

6 

8 

12 

IO 

5 

14 

6 

11 

19 

6 

5 

15 

9 

11 

19 

6 

4 

16 

6 

11 

19 

2 

5 

17 

6 

8 

19 

IO 

4 

18 

6 

11 

12 

2 

6 

19 

6 

8 

12 

6 

6 

20 

9 

8 

12 

2 

6 

21 

9 

11 

12 

2 

4 

22 

6 

11 

19 

6 

4 

23 

6 

11 

12 

6 

6 

24 

9 

11 

12 

6 

5 

25 

6 

11 

19 

10 

6 

26 

9 

11 

12 

2 

6 

27 

9 

8 

19 

6 

4 

28 

9 

8 

12 

2 

4 

29 

9 

11 

12 

IO 

5 

30 

6 

8 

19 

IO 

5 

31 

6 

11 

12 

6 

5 

32 

9 

8 

12 

6 

6 

33 

6 

8 

12 

IO 

4 

34 

6 

11 

12 

IO 

4 

35 

6 

11 

19 

IO 

4 

36 

6 

8 

12 

IO 

6 

37 

6 

11 

19 

IO 

5 

38 

9 

8 

12 

IO 

5 

39 

9 

8 

12 

IO 

6 

40 

6 

8 

12 

2 

4 

41 

9 

8 

19 

2 

4 

42 

9 

11 

12 

IO 

4 

43 

9 

11 

19 

2 

5 

44 

9 

8 

19 

IO 

6 

45 

6 

8 

19 

2 

5 

46 

6 

11 

12 

6 

4 

47 

9 

8 

12 

6 

5 

48 

6 

8 

12 

6 

5 

49 

6 

11 

19 

2 

4 

50 

9 

11 

12 

2 

5 

51 

9 

8 

12 

6 

4 

52 

6 

8 

12 

6 

4 

53 

9 

11 

19 

6 

6 

54 

9 

11 

19 

IO 

4 

55 

6 

8 

19 

6 

5 

56 

6 

11 

12 

2 

4 

57 

6 

11 

12 

IO 

5 

58 

6 

8 

19 

IO 

6 

59 

9 

11 

19 

2 

4 

60 

6 

8 

12 

2 

6 

61 

9 

11 

12 

10 

6 

62 

9 

8 

19 

2 

5 

63 

6 

8 

12 

2 

5 

64 

9 

8 

19 

6 

6 

65 

6 

8 

19 

2 

4 

66 

9 

11 

19 

IO 

5 

67 

6 

11 

19 

2 

6 

68 

9 

11 

19 

10 

6 

69 

9 

11 

12 

6 

6 

70 

9 

11 

19 

2 

6 

71 

9 

8 

19 

10 

4 

72 

6 

8 

19 

6 

6 
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IV.  ANALYSIS 


This  chapter  analyzes  the  results  of  all  three  experiments  and  points  to  a  preferred 
set  of  conditions  and  action  that  can  be  implemented  to  help  reduce  TTT.  The  analysis  is 
performed  using  the  JPM  statistical  package.  The  data  are  analyzed  using  partition  trees 
and  linear  regression  to  explain  the  hierarchical  relationship  between  the  factors  of 
interest  and  their  effect  on  TTT. 

A.  AIRCRAFT  UPGRADE  EFFECTS  ON  TTT 

The  experiment  to  determine  effects  of  aircraft  upgrades  on  TTT  is  run  twice. 
The  first  run  had  two  students  arriving  with  each  class.  This  number  of  students,  coupled 
with  the  induction  schedule  of  IPs,  aircraft,  and  FMS,  did  not  stress  the  system  enough  to 
make  any  significant  difference  in  TTT.  With  so  few  students  onboard,  the  resources 
were  adequate  to  cover  the  entire  range  of  noise  and  signal  variables.  TTT  remained 
fairly  constant  and  did  not  vary  by  more  than  10%  from  any  run.  The  utilization  rate  for 
the  FMS  was  less  than  10%  and  the  flight  hours  per  month  for  the  IPs  were  similarly 
around  10  hours  per  month,  which  is  below  the  recommended  proficiency  minimum. 
This  means  IPs  did  not  fly  enough  flight  events  to  satisfy  the  required  minimum  flight 
hours  and  would  be  required  to  fly  non- student-related  flights.  This  is  inefficient,  since 
these  extra  flights  are  not  flown  in  conjunction  with  student  events.  To  solve  this 
problem,  a  second  experiment  is  run  with  the  number  of  students  per  class  set  to  six, 
using  the  same  design  matrix. 

The  average  utilization  rate  for  FMS  in  this  new  design  is  32%,  while  the  average 
IP  flight  hours  increased  to  15  hours  per  month.  For  TTT,  the  result  of  this  new  design 
shows  it  is  insensitive  to  which  upgrade  scenario  is  chosen.  Table  9  shows  how  the  mean 
TTT  for  all  scenarios  are  statistically  equal  at  a  95%  confidence  level. 
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Table  9.  Analysis  of  Variance  of  Mean  Time  to  Train  of  Upgrade  Scenario 


Means  for  Oneway  Anova 


Level 

Number 

Mean 

Std  Error 

Lower  95% 

Upper  95% 

A 

99 

208.469 

1.6997 

205.13 

211.81 

A0 

33 

206.703 

2.9440 

200.92 

212.49 

B 

99 

208.624 

1.6997 

205.28 

211.97 

C 

99 

208.342 

1.6997 

205.00 

211.68 

D 

99 

208.373 

1.6997 

205.03 

211.71 

Std  Error  uses  a  pooled  estimate  of  error  variance 


Because  there  is  so  little  variability  in  mean  TTT,  a  formal  robust  analysis  of  the 
result  would  prove  fruitless  and  is  not  attempted  for  this  experiment.  However,  it  is 
interesting  to  note  that  the  default  scenario  considered  by  VMFAT-501  of  having  four 
aircraft  at  all  times  for  the  flight  schedule,  mod  C,  does  minimize  the  TTT  and  the 
standard  error  in  all  scenarios  other  than  the  control  group,  AO.  Therefore,  it  is  the  most 
robust  CO  A. 


Further  analysis  of  the  TTT  data  for  this  experiment  shows  a  nonnormal 
distribution  of  the  TTT  (see  Figure  7).  There  are  two  peaks  in  the  distribution,  a  lower 
peak  of  approximately  190  days  and  a  high  peak  of  approximately  225  days.  These  peaks 
can  be  explained  by  looking  at  the  partition  tree  of  the  data  by  the  noise  variables. 


170  180  190  200  210  220  230  240  250  260 


Figure  7.  Upgrade  Experiment  Histogram  of  Days  to  Train 


The  majority  of  the  cases  with  TTT  in  the  left  peak  have  refly  rates  less  than  12% 
and  four  or  more  MOA  slots.  While  the  right  peak  is  composed  of  cases  where  the  refly 
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rate  is  equal  to  or  greater  than  12%  and  aircraft  failure  rates  at  or  above  13%.  These 
factors  and  levels  should  be  closely  monitored  in  order  to  anticipate  and  rectify 
possible  delays. 


1.  Partition  Tree  of  Upgrade  Data 

Another  way  to  visualize  the  important  factors  in  determining  TTT  is  through  a 
partition  tree.  This  method  of  modeling  splits  the  data  into  groups  so  as  to  minimize  the 
total  sum  of  squares  error  (SSE).  The  factor  and  level  of  the  factor  to  split  is  chosen  by 
the  algorithm,  based  on  which  maximizes  the  reduction  in  the  total  SSE.  In  other  words, 
the  data  is  partitioned  into  two  groups  according  to  which  factor  and  level  explains  the 
majority  of  the  variation  in  the  data.  In  JMP,  the  splits  are  controlled  manually  until  the 
desired  degree  of  accuracy  is  achieved  or  the  resulting  pool  of  data  has  reached  a 
minimum  required  for  further  splits  because  of  insufficient  observations  or  insufficient 
variability  in  the  response  (Sail,  Creighton,  &  Lehman,  2005).  The  partition  of  the 
upgrade  data  is  shown  in  Figure  8.  There  are  no  signal  factors  present  after  three  splits. 
It  is  not  until  split  26  that  the  choice  of  upgrade  scenario  becomes  a  factor  in  detennining 
TTT.  This  further  confirms  that  TTT  is  insensitive  to  the  chosen  upgrade  COA. 


Figure  8.  Partition  of  Upgrade  Data  Including  Signal  and  Noise  Factors 
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By  further  expanding  the  partition  tree,  the  upper  and  lower  tails  of  the 
distribution  can  be  explored.  Design  points  with  an  average  TTT  near  181  days  form  the 
lower  tail.  They  are  composed  of  situations  where  flight  refly  is  below  12%,  MOA  slots 
are  greater  than  four,  FMS  refly  is  below  14%,  winter  cancellation  rates  are  below  16%, 
and  there  are  1 1  or  more  target  slots  available  each  day.  The  upper  tail  is  composed  of 
situations  with  flight  refly  greater  than  12%,  aircraft  failure  greater  than  13%,  aircraft 
repair  time  greater  than  five  hours,  and  less  than  26  MOA  slots.  Looking  at  the  partition 
tree  with  the  noise  data  is  not  useful  since  these  are  things  that  cannot  be  changed. 
Looking  at  the  partition  only  as  a  function  of  the  signal  factors  will  also  be  of  little  use 
since  the  means  of  each  COA  are  so  close.  The  difference  in  each  COA  is  not  enough  to 
explain  the  variation  in  TTT.  When  it  comes  to  what  decisions  to  make  that  will  help 
reduce  TTT,  the  choice  of  what  upgrade  scenario  to  use  should  not  be  a  factor  to 
consider. 

B.  AIRSPACE  AND  TTT 

Airspace  is  an  important  factor  in  the  smooth  operation  of  an  ITC.  Results  from 
the  upgrade  experiment  shows  how  important  the  number  of  MOA  slots  is  in  reducing 
TTT.  The  design  described  in  Chapter  III  brings  the  other  two  elements  of  air  space, 
target  slots  and  low-level  routes,  into  a  full  experiment  to  explore  the  impact  of  these 
resources  on  TTT. 

Results  from  this  experiment  are  similar  to  those  for  the  upgrade  scenario.  There 
is  no  relationship  between  airspace  and  TTT.  This  can  be  seen  in  a  simple  plot  of  TTT 
versus  airspace  resources  (see  Figure  9). 
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mean.TTT 

—  mean.TTT 

—  mean.TTT 

—  mean.TTT 


Figure  9.  Airspace  versus  Time  to  Train  Experiment  1 


The  plot  is  fairly  stable,  with  the  average  TTT  around  260  and  with  very  little 
variation.  The  MOA  plot  shows  a  tendency  toward  higher  TTT  as  the  number  of  MOA 
slots  decrease.  This  behavior  is  expected,  but  not  present,  in  either  of  the  other  two  plots. 
Instead,  they  are  fairly  constant  except  for  two  excursions  above  the  mean  at  20  and  25 
and  15  and  20  for  the  target  and  low-level  plots,  respectively.  A  linear  regression  of  the 
data  verifies  what  the  plots  show — there  is  no  relationship  between  TTT  and  airspace 
(see  Figure  10). 
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Analysis  of  Variance 


Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

3 

1031.64 

343.879 

0.7474 

Error 

1085 

499241.64 

460.131 

Prob  >  F 

C.  Total 

1088 

500273.28 

0.5240 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

261.6333 

2.046956 

127.82 

<.0001* 

MOA.SIots 

-0.102315 

0.072236 

-1.42 

0.1569 

Low.  Level 

0.0152619 

0.072237 

0.21 

0.8327 

Target.  Slots 

0.0312555 

0.072236 

0.43 

0.6653 

Figure  10.  Airspace  Regression  Statistics 


The  F  statistic,  the  mean  square  of  the  model  divided  by  the  mean  square  of  the 
error,  shows  the  model  is  statistically  insignificant.  Mean  TTT  is  a  better  predictor  of 
actual  TTT  than  any  relationship  between  the  number  and  type  of  airspace  available.  The 
parameter  estimates  show  the  factors  have  no  significant  effect  on  TTT. 

To  verify  these  findings,  and  ensure  that  the  1  to  30  range  in  the  airspace  factors 
is  not  too  large  and  thus  masking  a  relationship  with  TTT,  a  second  experiment  is  run. 
The  parameters  remain  unchanged,  and  the  design  is  an  NOLH  with  65  design  points. 
The  airspace  factor  levels  in  the  second  experiment  are  reduced  by  a  factor  of  three. 
Airspace  values  can  now  range  from  1  to  10,  and  should  provide  greater  resolution  on  the 
effects,  if  any,  of  airspace  on  TTT.  To  reduce  the  amount  of  computation  required,  the 
number  of  repetitions  is  reduced  to  35  at  each  design  points.  Figure  1 1  shows  the  linear 
regression  of  the  second  experiment. 


Analysis  of  Variance 


Source 

DF 

Sum  of 
Squares 

Mean  Square 

F  Ratio 

Model 

3 

1027.380 

342.460 

0.8743 

Error 

58 

22719.563 

391.717 

Prob  >  F 

C.  Total 

61 

23746.943 

0.4598 

Parameter  Estimates 

Term  Estimate  Std  Error  t  Ratio  Prob>|t| 


Intercept 

253.56757 

9.5131 

26.65 

<.0001* 

MOA.SIots 

0.5077986 

0.940488 

0.54 

0.5913 

Low.  Level 

1.3412531 

0.955644 

1.40 

0.1658 

Target.Slots 

-0.621882 

0.953691 

-0.65 

0.5169 

Figure  1 1 .  Airspace  Regression  Reduced  Levels  Statistics 
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The  results  from  both  regressions  confirm  that  airspace  and  TTT  have  no 
relationship  in  the  present  scenario.  The  best  conclusions  made  from  this  experiment  is 
that  airspace  is  not  a  potential  bottleneck  in  the  system  and  any  change  short  of 
elimination  of  the  airspace  will  have  very  little  influence  on  TTT. 

C.  SYLLABUS  VARIABLES  AND  TTT 

The  data  from  this  experiment  are  analyzed  using  both  a  linear  regression  model 
and  a  partition  tree.  The  regression  produces  a  good  fit  to  the  data,  but  does  not  explain 
or  provide  a  set  of  rules  or  guidelines  for  the  commander  to  use.  Both  courses  of  analysis 
do  agree  on  what  factors  are  most  important.  The  partition  tree  is  the  best  model  because 
of  its  simplicity  to  view,  understand,  and  explain. 


1.  Partition  Tree 

Figure  12  shows  a  partial  regression  tree  to  three  levels.  This  tree  explains  the 
majority  of  the  variability  in  the  data,  with  a  resulting  R-squared  over  (0.99).  The  data 
used  on  the  partition  tree  includes  all  50  repetitions  of  the  original  72  design  points. 


Figure  12.  Partition  Tree  of  Syllabus  Data 

The  most  influential  factor  affecting  TTT  is  the  number  of  students  in  each  class. 
The  mean  TTT  with  2,  6,  or  10  students  per  class  is  78,  286,  and  702  days,  respectively. 
Because  of  this,  the  analysis  is  broken  down  into  three  scenarios  based  on  how  many 
students  are  in  the  classes. 
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Figure  13.  Partition  Tree  for  Two  Students  per  Class 


Scenario  1  is  shown  in  Figure  13.  If  there  are  only  two  students  per  class,  then  it 
is  recommended  to  have  their  workday  set  at  1 1  hours  or  more  and  start  at  0900  or  later. 
These  factors,  at  these  levels,  would  produce  a  TTT  of  approximately  67  days  with  a 
standard  deviation  (SD)  of  0.5  days.  Having  student  workdays  of  less  than  1 1  hours,  and 
starting  before  0900,  gives  a  mean  TTT  of  90  days  with  an  SD  of  0.6  days.  It  is 
interesting  to  note  that  if  the  number  of  FMSs  could  be  increased  for  this  group,  it  would 
only  decrease  TTT  by  no  more  than  0.5  days.  This  group  is  more  sensitive  to  start  times 
and  workday  lengths  than  to  the  number  of  FMSs,  and  adding  additional  FMSs  would  not 
help  to  significantly  reduce  the  TTT. 
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Figure  14.  Partition  Tree  for  Six  Students  per  Class 


Scenario  2  is  shown  in  Figure  14.  If  there  are  six  students,  the  best  setting  of 
factors  is  to  increase  the  student  workday  to  1 1  or  more  hours  and  have  them  start  at  0900 
or  later.  With  four  FMSs,  this  gives  a  TTT  of  280  days  with  an  SD  of  eight  days.  The 
effect  of  having  more  FMSs  is  now  significant.  Having  five  or  more  FMSs  would 
decrease  TTT  by  nine  days,  or  about  two  weeks  of  training  time.  This  benefit  may  justify 
the  additional  cost  of  more  FMS  resources.  Not  only  does  it  decrease  TTT,  but  the  SD  is 
also  reduced,  making  it  a  robust  decision.  Another  interesting  outcome  of  more  students 
is  the  increase  in  the  variability  of  their  TTT.  The  SD  has  gone  from  about  1%,  with  two 
students  per  class,  to  4%  of  mean  TTT,  with  six  students  per  class. 
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Figure  15.  Partition  Tree  for  10  Students  per  Class 

Any  desire  to  further  decrease  TTT  would  have  to  focus  on  other  resources  that 
are  limiting  production.  In  this  case,  FMS,  by  itself,  is  not  a  bottleneck  in  the  system. 
The  next  likely  bottleneck  may  be  from  the  number  of  IPs  or  training  aircraft. 

Scenario  3  is  shown  in  Figure  15.  If  there  are  10  or  more  students,  the  most 
important  factor  in  reducing  TTT  is  the  number  of  FMSs.  A  reduction  of  35  days  is 
achieved  by  having  five  or  more  FMSs,  along  with  reducing  the  SD  by  over  six  days. 

The  next  most  important  factor  is  increasing  student  start  time.  This  is  a  change 
from  the  last  two  scenarios,  where  student  workday  was  more  important  than  student  start 
time.  Having  students  start  at  0900  or  later  reduces  TTT  by  20  to  35  days,  depending  on 
the  number  of  FMSs  available. 

Of  note  is  how  little  effect  the  student  workday  day  has  on  the  overall  TTT.  In 
scenarios  one  and  two,  it  was  the  primary  factor  in  reducing  TTT.  With  10  or  more 
students,  the  advantages  of  longer  workdays  are  almost  insignificant.  In  one  instance, 
having  student  workdays  at  or  greater  than  1 1  hours  only  reduces  TTT  by  less  than  0.2 
days.  This  is  due  to  other  constraints  in  the  ITC  such  as  instructors  and/or  classroom 
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availability.  Even  with  longer  workdays,  the  student  is  unable  to  utilize  the  limited 
resources. 

Clearly,  having  students  take  almost  three  years  to  complete  training  is 
unacceptable.  No  levels  of  the  signal  factor  looked  at  in  this  experiment  can  decrease  the 
TTT  to  an  acceptable  level.  To  further  explore  the  best  ways  to  decrease  TTT  when  there 
are  10  or  more  students  arriving  with  each  class,  this  experiment  needs  to  be  expanded  to 
look  at  other  factors  in  the  model.  Possible  candidate  factors  are  number  of  aircraft, 
instructors,  runways,  and  airspace  available.  Each  of  these  factors  will  increase  the  TTT 
if  they  are  in  short  supply,  but  collectively,  as  a  group,  their  effects  are  magnified. 
Fortunately,  this  work  is  ongoing  and  research  already  begun  to  explore  these  additional 
factors  as  well  as  the  inclusion  of  noise  variables. 
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V.  RECOMMENDATIONS 


A.  BEST  COURSE  OF  ACTION 

The  following  recommendations  are  made  to  help  VMFAT-501  reduce  TTT  as 
they  stand  up  the  first  JSF  ITC  at  Eglin  Air  Force  Base.  These  recommendations  are 
based  on  analysis  of  three  experiments  over  a  finite  time  period  with  constrained 
resources,  and  are  described  in  Chapters  III  and  IV. 

The  choice  of  upgrade  scenarios  and  resulting  loss  of  training  aircraft  from  one  to 
three  weeks  have  very  little  effect  on  TTT  when  there  are  six  or  less  students  per  class. 
The  choice  of  using  four  aircraft  to  support  flight  operations,  while  excess  aircraft 
undergo  upgrades,  is  the  most  robust  choice  of  the  12  scenarios  looked  at  in  the 
experiment,  not  including  the  control  scenario.  This  is  the  current  scenario  VMFAT-501 
intends  to  implement.  Factors  to  watch  that  could  lead  to  increasing  TTT  are  having 
flight  and  FMS  refly  rates  above  12%  and  14%,  respectively,  and  winter  cancellation  and 
aircraft  failure  rates  above  16%  and  13%,  respectively.  All  factors  that  have  a  significant 
effect  on  TTT  are  uncontrollable,  and  must  be  continuously  monitored  in  order  to  have 
reasonable  estimates  and  to  anticipate  changes  to  a  student’s  TTT. 

Airspace  was  also  another  factor  that  had  little  effect  on  TTT.  The  availability  of 
airspace  for  training,  short  of  eliminating  this  resource,  did  not  present  a  bottleneck  to 
training  and  did  not  change  TTT. 

The  most  profound  recommendations  come  from  the  last  experiment.  The 
recommendations  all  depend  on  the  number  of  students  per  class.  As  student  numbers 
increase  from  2  to  10  students  per  class,  so  does  the  TTT.  Increasing  the  class  size 
directly  affects  TTT  if  the  resources  are  kept  constant.  As  the  number  of  students 
increases,  the  magnitude  of  the  effects  also  increases  and  the  hierarchy  of  significant 
factors  changes.  The  factors  do  not  have  the  same  weight  for  class  sizes  of  2,  6,  or 
10  personnel. 

Start  time  was  significant  in  all  three  cases,  as  detennined  by  the  partition  tree  in 
Chapter  III.  The  commander’s  best  decision,  regardless  of  the  number  of  students,  is  to 
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have  students  start  at  or  after  0900,  to  reduce  the  average  TTT.  This  reduces  TTT  by  2, 
4,  and  20  days  for  class  sizes  of  2,  6,  and  10  personnel,  respectively.  This  may  be  due  to 
bottlenecks  in  the  system  for  the  schedule  and  completion  of  night  events.  The  ITC  is 
very  strict  and  inflexible  when  it  schedules  students  for  events,  which  is  very  different 
from  how  a  squadron  runs  its  flight  schedule. 

Student  workday  length  was  significant  in  only  two  of  the  cases  tested.  With  10 
or  more  students,  the  length  of  a  student’s  workday  had  no  significant  effect  on  reducing 
the  TTT.  With  six  or  less  it  was  the  dominate  factor  in  determining  the  TTT,  reducing  it 
by  more  than  20  days  in  both  cases.  A  commander’s  best  decision  is  to  have  student 
workdays  of  1 1  hours  or  more,  if  there  are  six  or  less  students  per  class. 

FMSs  are  an  expensive  resources  and  their  effect  on  TTT  must  be  significant 
enough  to  justify  their  increased  expense.  The  number  of  FMSs  was  significant  in  only 
two  of  the  cases  tested.  With  two  students  per  class,  increasing  the  number  of  FMSs 
beyond  four  has  no  significant  contribution  to  decreasing  TTT.  With  six  students  per 
class,  the  TTT  only  decreased  by  nine  days,  or  almost  two  training  weeks  at  the  ITC, 
which  is  a  significant  amount  of  time.  When  compared  to  an  average  TTT  of  over  a  year, 
the  two  weeks  may  not  be  significant  enough  to  justify  more  FMSs;  rather,  there  may  be 
other  constraints  that  are  more  economically  feasible  and  easier  to  overcome.  When 
there  are  10  students  per  class,  the  reduction  in  TTT  is  greater  than  30  days,  which  is 
significant,  but  compared  to  the  two  plus  years  of  TTT,  it  is  not  enough  to  justify  more 
FMSs.  In  this  case,  the  increase  in  TTT  is  due  to  constraints  in  other  resources.  Potential 
sources  of  this  constraint  are  the  number  of  IPs,  aircraft,  classrooms,  or  runways  available 
for  training.  Interestingly,  ITC  operational  hours  have  no  effect  on  the  TTT. 

The  overall  recommendation  for  reducing  TTT  is  to  have  students  work  1 1  hours 
per  day  and  have  the  workday  start  at  or  after  0900.  Increasing  the  number  of  FMS  to 
five  or  six,  while  reducing  TTT,  is  not  justified.  Likewise,  increasing  the  hours  that  the 
ITC  remains  open  has  no  effect  on  reducing  TTT  and  should  be  ignored.  The  current  plan 
of  having  four  aircraft  reserved  for  training  events,  while  the  excess  undergo  upgrades,  is 
the  best  and  most  robust  COA  to  ensure  a  minimum  TTT. 
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The  results  produced  by  the  ITC  are  inherently  optimistic  due  to  deficiencies  in 
how  the  model  treats  weather,  maintenance,  other  stochastic  elements,  and  in  the  strict 
scheduling  of  student  events  (Duminar,  2011).  The  ITC  model  also  uses  means  rather 
than  distribution  for  its  random  variables,  this  also  gives  optimistic  results.  As  a 
consequence,  any  reductions  in  TTT  gained  for  choosing  the  COAs  recommended  in  this 
thesis  are  also  optimistic.  These  findings  do  provide  a  relationship  between  TTT  and  the 
controllable  and  uncontrollable  factors.  It  is  up  to  the  commander  and  training  officers  to 
monitor  these  factors  and  use  the  COAs  to  help  mitigate  their  effects. 

B.  FURTHER  STUDY 

The  metric  of  focus  has  been  TTT,  but  there  are  other  factors  that  could  also  be 
included  to  build  a  more  dynamic  model  of  the  process.  Flight  hours  per  aircraft  and  per 
IP  are  very  important  and  highly  correlated  to  the  TTT.  The  JSF,  like  any  other  machine, 
will  have  mechanical,  maintenance,  and  administrative  limits  driven  by  utilization  that 
cannot  be  breached.  Once  the  squadron  receives  its  first  aircraft,  it  will  also  receive  an 
allotment  of  funds  for  fuel  purchases  each  quarter.  This  allotment  and  the  current  cost  of 
fuel  will  determine  a  flight-hour  budget  and  a  maximum  total  number  of  flight  hours  per 
month  that  the  squadron  can  fly.  This  factor  is  not  simulated  in  the  ITC,  but  is  recorded 
as  part  of  the  output.  There  are  no  limitations  placed  on  how  many  hours  or  sorties  an 
aircraft  can  fly,  as  there  are  limits  on  how  many  hours  a  student  or  instructor  can  work  in 
a  day.  To  capture  this  effect,  an  experiment  would  have  to  be  designed  to  look  at  aircraft 
utilization  for  each  month,  coupled  with  a  predicted  flight-hour  budget,  to  see  which 
scenarios  are  fiscally  possible  and  to  help  predict  what  level  of  funding  to  request  for 
training.  This  can  also  help  predict  a  more  reliable  estimate  of  TTT  by  ruling  out 
infeasible  scenarios. 

An  investigation  of  other  resources,  not  explored  in  this  thesis,  may  be  helpful  in 
discovering  other  bottlenecks  in  the  system.  A  study  of  the  delays  in  completing  events 
could  help  pinpoint  a  general  direction  in  which  to  find  and  fix  these  bottlenecks.  The 
explanations  of  these  delays  are  often  vague  and  may  require  multiple  designs  to  properly 
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investigate  them.  Combining  designs  of  experiment  one,  two,  and  three  into  one  grand 
design  may  produce  a  better  understanding  of  the  relationship  between  all  factors. 

This  thesis  looked  at  the  ITC  model  during  the  ramp-up  mode.  Further  studies  in 
the  steady  state  mode  would  be  of  value,  especially  once  reliable  statistics  become 
available  with  respect  to  aircraft  failure  rates  and  repair  times,  as  well  as  student  attrition 
rates  and  weather  cancellation  rates.  Without  actual  data  on  JSF  training  times,  the  ITC 
model  can  continue  to  be  a  reliable  predictor  and  tool  for  the  commander  when  faced 
with  unforeseen  changes  in  resources  or  unexplained  changes  in  some  of  the  noise 
factors. 
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